Image capture control methods and apparatus

ABSTRACT

Methods and apparatus for controlling the read out of rows of pixel values from sensors corresponding to different optical chains used to capture portions of the same image area are described. The readout is controlled based on user input and/or determinations with regard to the rate of motion in captured images or portions of captured images. For a low rate of motion i, the readout rate of a sensor corresponding to a small focal length is slowed down while the pixel row readout rate of one or more sensors corresponding to one or more optical chains have larger focal lengths are allowed to proceed at a normal rate. For a high rate of motion, the read out rate of the sensor corresponding to the optical chain having the smaller focal length is allowed to proceed at the normal rate.

RELATED APPLICATIONS

The present application claims the benefit of U.S. ProvisionalApplication Ser. No. 61/978,818 filed Apr. 11, 2014, U.S. ProvisionalApplication Ser. No. 61/981,849, filed Apr. 20, 2014, and U.S.Provisional Application Ser. No. 62/021,094 filed Jul. 4, 2014, each ofwhich is hereby expressly incorporated by reference in their entirety.

FIELD

The present application relates to image capture and generation methodsand apparatus and, more particularly, to methods and apparatus relatedto control of camera devices including multiple optical chains and/orwhich process the output of multiple optical chains of a camera device,e.g., to generate one or more composite images.

BACKGROUND

High quality digital cameras have to a large extent replaced filmcameras. However, like film cameras, with digital cameras much attentionhas been placed by the camera industry on the size and quality of lenseswhich are used on the camera. Individuals seeking to take qualityphotographs are often encouraged to invest in large bulky and oftencostly lenses for a variety of reasons. Among the reasons for usinglarge aperture lenses is their ability to capture a large amount oflight in a given time period as compared to smaller aperture lenses.Telephoto lenses tend to be large not only because of their largeapertures but also because of their long focal lengths. Generally, thelonger the focal length, the larger the lens. A long focal length givesthe photographer the ability to take pictures from far away.

In the quest for high quality photos, the amount of light which can becaptured is often important to the final image quality. Having a largeaperture lens allows a large amount of light to be captured allowing forshorter exposure times than would be required to capture the same amountof light using a small lens. The use of short exposure times can reduceblurriness especially with regard to images with motion. The ability tocapture large amounts of light can also facilitate the taking of qualityimages even in low light conditions. In addition, using a large aperturelens makes it possible to have artistic effects such as small depth offield for portrait photography.

While large lenses have many advantages with regard to the ability tocapture relatively large amounts of light compared to smaller lenses,they can be used to support large zoom ranges which may be implementedusing optical or digital techniques, and often allow for good controlover focus, there are many disadvantages to using large lenses.

Large lenses tend to be heavy requiring relatively strong and oftenlarge support structures to keep the various lenses of a camera assemblyin alignment. The heavy weight of large lenses makes cameras with suchlenses difficult and bulky to transport. Furthermore, cameras with largelenses often need a tripod or other support to be used for extendedperiods of time given that the sheer weight of a camera with a largelens can become tiresome for an individual to hold in a short amount oftime.

In addition to weight and size drawbacks, large lenses also have thedisadvantage of being costly. This is because of, among other things,the difficulty in manufacturing large high quality optics and packagingthem in a manner in which they will maintain proper alignment over aperiod of time which may reflect the many years of use a camera lensesis expected to provide.

In digital cameras, the photosensitive electronics used as the sensor,e.g., light sensing device, is often either a charge coupled device(CCD) or a complementary metal oxide semiconductor (CMOS) image sensor,comprising a large number of single sensor elements, each of whichrecords a measured intensity level.

In many digital cameras, the sensor array is covered with a patternedcolor filter mosaic having red, green, and blue regions in anarrangement. A Bayer filter mosaic is one well known color filter array(CFA) for arranging RGB color filters on a square grid of photo sensors.Its particular arrangement of color filters is used in many digitalimage sensors. In such a filter based approach to capturing a colorimage, each sensor element can record the intensity of a single primarycolor of light. The camera then will normally interpolate the colorinformation of neighboring sensor elements, through a process sometimescalled demosaicing, to create the final image. The sensor elements in asensor array using a color filter are often called “pixels”, even thoughthey only record 1 channel (only red, or green, or blue) of the finalcolor image due to the filter used over the sensor element.

In cameras, round or circular lenses (lens element) through which lightcan pass, e.g., lenses with round apertures, are commonly used. Thisallows light to pass through the lens equaling in both vertical andhorizontal directions (actually any direction). Elements in a opticalchain in which one or more round lenses are used are often of sufficientsize to pass enough light to record a sharp image at the sensor of thecamera. For a lens, the optical axis is the line passing through thecenter of the lens and perpendicular to the plane of the lens. When alens assembly is constructed out of more than one lens element, theelements are typically arranged so that they all share a common opticalaxis which is also the optical axis of the lens assembly. In a typicaloptical chain, e.g., camera module, the optical axis also passes throughthe center of the sensor. Light traveling along the optical axis of alens assembly or camera module is not bent by any lens along the pathand continues to travel along the optical axis in a straight line. Ifall the lens elements are circular, then such a camera module hascylindrically symmetry around the optical axis. In most cameras, theoptical elements of a camera are arranged in a linear configuration withoptical axis passing through an outer lens in a straight line to thesensor. Such a configuration can result in relatively thick cameras,e.g., cameras having a large front to back distance or depth. In cameraswith large optics and/or which support mechanical zoom, the camerathickness can be significant with the camera often being several inchesthick and far too deep to store in a pocket or even in some cases apurse.

Cameras and other optical systems are often discussed in terms of focallength. The focal length of an optical system is a measure of howstrongly the system converges or diverges light. For an optical systemin air, it is the distance over which initially collimated rays arebrought to a focus. A system with a shorter focal length has greateroptical power than a system with a long focal length; that is, it bendsthe rays more strongly, bringing them to a focus in a shorter distance.Longer focal length (lower optical power), often achieved using largelenses, leads to higher magnification, e.g., zoom, and a narrower angle(field) of view. Accordingly, an optical chain, e.g., camera module,with a large, e.g., long, focal length will capture an imagecorresponding to a smaller portion of a scene area than an optical chainat the same location with a smaller focal length. It should beappreciated that for the same sensor size, an optical chain with ashorter focal length or higher optical power is associated with a widerangle of view than an optical chain with a longer focal length and willthus capture an image corresponding to a larger portion of a scene areaat the same distance from the optical chain than an optical chain at thesame position with a larger focal length.

Focal length of a lens element is generally not a function of lens size,e.g., diameter in the case of lens element lenses with round apertures,e.g., round areas through which light can pass. In an optical chain orother optical device the focal length of the device, is sometimesreferred to as the effective focal length, since the focal length of thedevice will depend on the one or more optical elements, e.g., lenses,which make up the device and their interaction. To take good qualitysharp pictures requires larger aperture lens elements (larger diameterlenses) when the effective focal length is large.

The use of large focal length lenses and optical chains with large focallengths is often desirable because of the amount of zoom (magnification)that they can provide. However, the use of optical components which arecommonly used to produce an optical chain with a large focal lengthtends to lead to a thick camera particularly in the case where theoptical components are arranged so that the light along the optical axispasses in a straight line from the outermost lens to the sensor which isused to capture an image based on the light passing through a lens.

From the above discussion is should be appreciated that there is a needfor new or improved methods and/or apparatus which provide for controlof a camera device, e.g., to facilitate capture of images by one or moreoptical chains of a camera device in accordance with user input and/oran automated selection of a mode of operation where the mode ofoperation may determine when or how pixel values are read out of one ormore sensors.

SUMMARY

Methods and apparatus for controlling the read out of rows of pixelvalues from sensors corresponding to different optical chains used tocapture portions of the same image area are described. The readout may,and in some embodiments is controlled based on user input and/ordeterminations with regard to the rate of motion in captured images orportions of captured images. In some embodiments, in cases where therate of motion is low, the readout rate of a sensor corresponding to asmall focal length is slowed down while the pixel row readout rate ofone or more sensors corresponding to one or more optical chains havelarger focal lengths are allowed to proceed at a normal rate. In someembodiments, in cases where a high rate of motion is detected as may be,and in some embodiments is, determined from a large difference between afirst image and a second image captured at a later time, the read outrate of the sensor corresponding to the optical chain having the smallerfocal length and thus capturing the larger image area is allowed toproceed at the normal rate. In at least some embodiments where thesensor corresponding to the optical chain is read out at the normal rateone or more sensors corresponding to optical chains having a largerfocal length are controlled to perform joint readouts of rows of pixelvalues thereby allowing the readout of the sensors corresponding tomultiple optical chains having the higher focal length to be completedin the same time or approximately the same time as the time required toread out the sensor corresponding to the optical chain having the smallfocal length. The optical chains corresponding to the larger focallength may correspond to different image portions and be read outsequentially or at least in a manner where portions of the sensors areread out at different times allowing for the synchronization of thereadout of the sensor corresponding to the small focal length opticalchain with the readout the sensors corresponding to the larger focallength optical chains in a manner which minimizes or reduces temporaldistortions which might otherwise be introduced into a composite imagegenerated by combining the images captured by different optical chains.

An exemplary method, in accordance with some embodiments, of capturingimages using a camera device including at least a first optical chainincluding a first sensor and a second optical chain including a secondsensor, said first optical chain having a smaller focal length than saidsecond optical chain, comprises: making a decision based on at least oneof: i) user input or ii) a determined rate of motion whether to controlthe second sensor of the second optical chain to perform joint readoutsof pixel elements corresponding to consecutive pixel element rows of thesecond sensor or to perform individual read outs of consecutive rows ofpixel elements of said second sensor; controlling the first sensor toreadout pixel values of said first sensor; and controlling the secondsensor to readout pixel values of pixel elements of said second sensorin accordance with the said decision to perform joint readouts of pixelelements corresponding to consecutive pixel element rows of the secondsensor or to perform individual read outs of consecutive rows of pixelelements of said second sensor.

An exemplary camera device, in accordance with some embodiments,includes: a first optical chain including a first sensor and having afirst focal length; a second optical chain including a second sensor andhaving a second focal length, said first focal length being smaller thansaid second focal length; a readout control decision module configuredto make a readout control decision, based on at least one of: i) userinput or ii) a determined rate of motion, whether to control the secondsensor of the second optical chain to perform joint readouts of pixelelements corresponding to consecutive pixel element rows of the secondsensor or to perform individual read outs of consecutive rows of pixelelements of said second sensor; and a sensor readout control moduleconfigured to: i) control the first sensor to readout pixel values ofsaid first sensor; and ii) control the second sensor to readout pixelvalues of pixel elements of said second sensor in accordance with thesaid readout control decision.

A camera device, in accordance with some embodiments, includes anon-transitory machine readable medium, the non-transitory machinereadable medium including processor executable instructions which whenexecuted by a processor of the camera device including at least a firstoptical chain including a first sensor and a second optical chainincluding a second sensor, said first optical chain having a smallerfocal length than said second optical chain, control the camera deviceto perform the steps of: making a decision based on at least one of: i)user input or ii) a determined rate of motion whether to control thesecond sensor of the second optical chain to perform joint readouts ofpixel elements corresponding to consecutive pixel element rows of thesecond sensor or to perform individual read outs of consecutive rows ofpixel elements of said second sensor; controlling the first sensor toreadout pixel values of said first sensor; and controlling the secondsensor to readout pixel values of pixel elements of said second sensorin accordance with the said decision to perform joint readouts of pixelelements corresponding to consecutive pixel element rows of the secondsensor or to perform individual read outs of consecutive rows of pixelelements of said second sensor.

While many embodiments and features have been discussed in the abovesummary, it should be appreciated that many of the novel featuresdescribed herein can be used independent of other novel features. Thuswhile various exemplary embodiments have been described, the applicationis not limited to the particular exemplary embodiments or combinationsof features described in particular exemplary embodiments.

Numerous additional features and embodiments are described in thedetailed description which follows.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of an exemplary apparatus, e.g., a cameradevice, implemented in accordance with one embodiment of the presentinvention.

FIG. 2 illustrates a frontal view of an apparatus implemented inaccordance with an exemplary embodiment which incorporates multipleoptical chains, e.g., camera modules, in accordance with the presentinvention with lenses which are viewable from the front of the camera.

FIG. 3, which is a side view of the exemplary apparatus of FIG. 2,illustrates further details of the exemplary apparatus.

FIG. 4A illustrates a camera device implemented in accordance withanother embodiment.

FIG. 4B illustrates the optical chains of the camera device shown inFIG. 4A, as implemented in one particular exemplary embodiment, ingreater detail.

FIG. 5 illustrates an exemplary optical chain, e.g., camera module,which may be used as one of the optical chains included in the cameradevice of FIG. 1, FIG. 8, FIG. 12A or various other embodiments.

FIG. 6 is a prospective view of an exemplary camera including multiplecamera modules in accordance with one feature of the invention with itscover retracted to allow the camera modules to capture images.

FIG. 7 is illustrates the camera of FIG. 6 with the cover in closedposition thereby covering the camera front.

FIG. 8 is a frontal view of a camera device similar to the one of FIG. 6with a better view of the camera modules so that the arrangement of thelenses of the individual camera modules can be better appreciated.

FIG. 9 is an illustration of the camera device of FIG. 8 but without thecamera case being shown allowing for better appreciation of thearrangement of lenses on the front of the camera device.

FIG. 10 is a side view of the camera device of FIG. 7 from which it canbe seen that the camera has a depth D1 and a height H.

FIG. 11 illustrates the arrangement of lenses shown in FIGS. 8 and 9 ingreater detail.

FIG. 12A illustrates an arrangement of optical chains, e.g., cameramodules, used in one embodiment to implement a camera device of the typeshown in FIGS. 6 and 8 with the lens arrangement shown in FIG. 11.

FIG. 12B illustrates a perspective view of a camera device of the typeshown in FIG. 8, with the arrangement of various optical chains andelements of the optical chains in the camera device shown in greaterdetail.

FIGS. 12C, 12D, 12E and 12F illustrate how a first portion of an opticalaxis of an optical chain shown in FIG. 12B can be changed by alteringthe position, e.g., angle, of a light redirection device which is partof the optical chain.

FIG. 13A shows a camera module with a lens that has a non-circularaperture and a light redirection device, e.g., mirror, which can be andis used in various camera device embodiments shown in FIGS. 14-17.

FIG. 13B is a drawing illustrating an exemplary arrangement of aplurality of optical chains (OCs) in a camera and the configuration andarrangement of the elements of each of the optical chains, in accordancewith an exemplary embodiment.

FIG. 14 illustrates an exemplary camera device which uses non-circularlenses for the optical chains having large focal lengths and circularlenses for optical chains which have smaller focal lengths.

FIG. 15 is a diagram of the front of the camera device shown in FIG. 14without the camera case.

FIG. 16 is a side view of the camera device of FIG. 14 from which it canbe seen that the camera has a depth D2 and a height H, where D2 is lessthan D1 which is the depth of the camera device shown in FIGS. 6-9.

FIG. 17A illustrates an arrangement of camera modules used in oneembodiment to implement a camera device of the type shown in FIG. 14with the lens arrangement shown in FIG. 15.

FIG. 17B illustrates a perspective view of a camera device of the typeshown in FIG. 14, with the arrangement of various optical chains andelements of the optical chains in the camera device shown in greaterdetail.

FIG. 18 shows a round aperture corresponding to an exemplary lens with around opening such as the lenses which may and sometimes are used in theFIG. 12A embodiment.

FIG. 19 shows the frequency characteristics which are expected from alens of the type shown in FIG. 18 with the frequency information beingthe same or similar in both dimensions.

FIG. 20 shows how, in the case of a round aperture, the length of theopening through which light passes is the same in both dimensions of theplane in which the lens opening exists.

FIG. 21 shows an exemplary non-round, e.g., oval, aperture with theshading used to show the relative amount of frequency information ineach of the horizontal and vertical directions which will be capturedwith it being clear from the figure that more frequency information isavailable in the vertical direction than in the horizontal directionthereby resulting in higher frequency information being captured andavailable in the longer dimension of the aperture than in the narrowerdimension.

FIG. 22 shows a comparison of the lengths of the non-round aperture inthe vertical (Y) and horizontal (X) directions, with the verticaldimension being the longer of the two dimensions in the FIG. 22 example.

FIG. 23 shows how, by combining image information from multiplenon-round lenses oriented in different directions, image informationapproximating the information expected to be obtained from a round lenscan be achieved with more information being available towards the centerof the combined image than at various edge locations due to theoverlapping of multiple individual images which are combined in the FIG.23 example to generate a composite image.

FIG. 24 shows how a lens with a round aperture can be cut or masked toproduce a lens having a non-round aperture, e.g., approximating that ofan oval or oblong shape.

FIG. 25 shows the aperture resulting form cutting or masking a roundlens, e.g., a lens with a round aperture, as shown in FIG. 24.

FIG. 26 shows how using a light redirection device to redirect light 90degrees in combination with an outer lens having a round aperturenormally requires a camera depth, e.g., thickness, equal to or greaterthan the diameter of the lens with the round aperture.

FIG. 27 shows how, in some embodiments, use of a lens with a non-round,e.g., oval, aperture in combination with a light redirection devicewhich redirects light by 90 degrees can allow for use of lenses whichare longer in one dimension than the camera is deep.

FIG. 28 shows how the length of the light path of a camera moduleincluding a non-round aperture can be longer than the depth of thecamera with the light redirection device being capable of beingpositioned at one end of the camera device.

FIG. 29 shows an example in which multiple light redirection devices areused in a camera module to allow for a relatively long light travel pathand thus focal length while allowing the sensor to be positioned oneither the back or front of the camera depending on which way the lightis redirected as it passes through a camera module.

FIG. 30 shows how multiple lenses with non-circular apertures can beused in a single exemplary camera device to collect high frequencyinformation in multiple directions so that high frequency information isavailable in each of a plurality of directions when combining images togenerate a composite image.

FIG. 31 shows an exemplary scene including a scene area which may haveits image captured by camera modules of a camera implemented inaccordance with one or more embodiments of the invention.

FIG. 32 shows how different camera modules of a camera includingmultiple camera modules, some of which have different focal lengths, maycapture different size portions of a scene area of interest.

FIG. 33 shows how different camera modules of a camera includingmultiple camera modules as shown in FIG. 32 may capture differentportions of the scene area of interest shown in FIG. 31.

FIG. 34 shows different image captured by a camera having modulescapturing scene areas of the sizes shown in FIG. 33 and theirrelationship to one another which facilitates combing of the images togenerate a composite image.

FIGS. 35, 36, 37, 38 and 39 show various aspects relating to rollingshutter control of the reading of sensors of different optical chains ina coordinated manner so that the images captured by the differentsensors can be easily combined.

FIG. 40 is an exemplary method of capturing images using multiple cameramodules and combining the images in accordance with one exemplaryembodiment.

FIG. 41, which comprises the combination of FIG. 41A and FIG. 41B, is aflow chart showing the steps of an exemplary method of operating acamera device to scan a scene of interest and generate a composite imagethere from.

FIG. 42 shows the steps of a method of capturing a scene of interestusing a plurality of camera modules in a synchronized manner.

FIG. 43 shows the steps of a method of capturing a scene of interestusing a plurality of camera modules in a synchronized manner inaccordance with another exemplary embodiment.

FIG. 44 shows the steps of a method of capturing images using aplurality of camera modules in a synchronized manner in accordance withan exemplary embodiment.

FIG. 45 is a flowchart illustrating a method of capturing images using acamera device in accordance with an exemplary embodiment.

FIG. 46 is a flowchart illustrating a method of capturing images inaccordance with an exemplary embodiment.

FIG. 47 is a flowchart of a method of generating a composite image inaccordance with an exemplary embodiment.

FIG. 48A is a drawing of an exemplary camera device including moveablemirrors in accordance with an exemplary embodiment.

FIG. 48B is a drawing of an exemplary camera device including moveableprisms in accordance with an exemplary embodiment.

FIG. 48C is a drawing of an exemplary assembly of modules, which may beincluded in an exemplary camera device in accordance with an exemplaryembodiment.

FIG. 49 is a drawing used to illustrate exemplary captured images usedto generate a combined image in accordance with an exemplary embodiment.

FIG. 50 is a drawing used to illustrate exemplary captured images usedto generate a combined image in accordance with an exemplary embodiment.

FIG. 51A is a first part of a flowchart of an exemplary method ofcapturing at least one image in accordance with an exemplary embodiment.

FIG. 51B is a second part of a flowchart of an exemplary method ofcapturing at least one image in accordance with an exemplary embodiment.

FIG. 52 is a drawing of an exemplary camera device including moveablemirrors in accordance with an exemplary embodiment.

FIG. 53 is a drawing of an exemplary camera device including moveableprisms in accordance with an exemplary embodiment.

FIG. 54 is a drawing of an exemplary assembly of modules, which may beincluded in an exemplary camera device in accordance with an exemplaryembodiment.

FIG. 55 is a drawing of an exemplary stored zoom setting to lightredirection devices positions mapping table in accordance with anexemplary embodiment.

FIG. 56 is a drawing illustrating an example in which a combined imageis generated from captured images from multiple optical chains,corresponding to a zoom setting of 1×, in accordance with an exemplaryembodiment.

FIG. 57 is a drawing illustrating an example in which a combined imageis generated from captured images from multiple optical chains,corresponding to a zoom setting of 2×, in accordance with an exemplaryembodiment.

FIG. 58 is a drawing illustrating an example in which a combined imageis generated from captured images from multiple optical chains,corresponding to a zoom setting of 3×, in accordance with an exemplaryembodiment.

FIG. 59 is a drawing illustrating an example in which a combined imageis generated from captured images from multiple optical chains,corresponding to a zoom setting of 4×, in accordance with an exemplaryembodiment.

FIG. 60A is a first part of a flowchart of an exemplary method ofcapturing images in accordance with an exemplary embodiment.

FIG. 60B is a second part of a flowchart of an exemplary method ofcapturing images in accordance with an exemplary embodiment.

FIG. 60C is a third part of a flowchart of an exemplary method ofcapturing images in accordance with an exemplary embodiment.

FIG. 60D is a fourth part of a flowchart of an exemplary method ofcapturing images in accordance with an exemplary embodiment.

FIG. 61 is a drawing of an exemplary camera device including a pluralityof optical chains with moveable light redirection devices in accordancewith an exemplary embodiment.

FIG. 62 is a drawing of an assembly of modules which may be included inan exemplary camera device in accordance with an exemplary embodiment.

FIG. 63 is a drawing which illustrates exemplary composite images,corresponding to different zoom focal length settings, in accordancewith an exemplary embodiment.

FIG. 64 illustrates captured images which are combined to generate acomposite image for an example in which the user selected focal lengthsetting is set at a maximum zoom out, e.g., zoom focal lengthsetting=f2, in accordance with an exemplary embodiment.

FIG. 65 illustrates captured images and cropped captured images, some ofwhich are combined to generate a composite image for an example in whichthe user selected focal length setting is set at a maximum zoom in,e.g., zoom focal length setting=f1, in accordance with an exemplaryembodiment.

FIG. 66 illustrates captured images and cropped captured images, some ofwhich are combined to generate a composite image for an example in whichthe user selected focal length setting is set at an intermediate value,e.g., a value between f2 and f1, in accordance with an exemplaryembodiment.

FIG. 67 illustrates exemplary user controlled zoom and exemplary overlapbetween captured images from different optical chains, each including amoveable light redirection device which is positioned, e.g., adjusted,as a function of a user selected zoom focal length setting, inaccordance with an exemplary embodiment.

FIG. 68A is a first part of a flowchart of an exemplary method ofcapturing an image using a camera device in accordance with an exemplaryembodiment.

FIG. 68B is a second part of a flowchart of an exemplary method ofcapturing an image using a camera device in accordance with an exemplaryembodiment.

FIG. 69 is a drawing of an exemplary camera device including at leasttwo optical chains with different focal lengths in accordance with anexemplary embodiment.

FIG. 70 is a drawing of an assembly of modules which may be included inan exemplary camera device in accordance with an exemplary embodiment.

FIG. 71 is a drawing illustrating an exemplary scene of interest andexemplary capture areas corresponding to different optical chains in acamera device in accordance with an exemplary embodiment.

FIG. 72 is a drawing illustrating exemplary readout times for an examplecorresponding to user input high rate of motion mode of operation orautomatic mode of operation with a high sensed rate.

FIG. 73 is a drawing illustrating exemplary readout times for an examplecorresponding to user input high resolution mode of operation orautomatic mode of operation with a low sensed rate.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary camera device 100 such as a digitalcamera, notepad with camera functionality, or cell phone with camerafunctionality, implemented in accordance with one exemplary embodimentof the present invention. The camera device 100, in some embodiments, isa portable device. In other embodiments, the camera device 100 is afixed device such as a wall mounted camera.

FIG. 1 illustrates the camera device 100 in block diagram form showingthe connections between various elements of the apparatus 100. Theexemplary camera device 100 includes a display device 102, a lightemitter module 104, an input device 106, an input state detection module148, an exposure and readout controller 150, e.g., a rolling shuttercontroller 150, a light control device 152, memory 108, a processor 110,a hardware assembly of modules 180, a wireless and/or wired interface114, e.g., a cellular interface, a WIFI interface, and/or a USBinterface, an I/O interface 112, an accelerometer module 122, 3 axisgyro 192, and a bus 116 which are mounted in a housing represented bythe rectangular box touched by the line leading to reference number 100.The light emitter module 104 includes light emitting elements whichmaybe LEDs (Light Emitting Diodes) or other types of light emittingelements which can be individually controlled so that all the lightemitting elements need not be on at the same time. The input device 106may be, and in some embodiments is, e.g., keypad, touch screen, orsimilar device that may be used for inputting information, data and/orinstructions. The accelerometer module 122 includes accelerometer 1 124,accelerometer 2, 126 and accelerometer 3 128 which are arrayed onperpendicular axis providing a 3 axis accelerometer module. Thus, theaccelerometer module 122 can measure along 3 independent axis.Similarly, the 3-axis gyro 192, which includes 194, 196 and 198 canmeasure rotation along each of 3 different axis. The output of theaccelerometer module 122 and the gyro module 192 can, and in someembodiments is, monitored with changes in accelerometer and gyro outputbeing interpreted and checked over time by processor 110 and/or zoomcontrol module, e.g., zoom controller 140 to detect changes inacceleration indicating motion in one or more directions. In someembodiments the input device 106 includes at least one zoom controlbutton that can be used to enable or disable camera zoom functionality.In some such embodiments when the zoom control button is in a depressedstate the camera zoom function is enabled while when the button is in aun-depressed state the camera zoom function is disabled. The input statedetection module 148 is configured to detect the state of the inputdevice, e.g., the zoom control button, to detect whether the button isin a depressed state or undepressed state. In some embodiments there isa status register in the camera device 100 that includes a bitindicating the state of the zoom control button detected by the statedetection module 148, e.g., whether it is in the depressed stateindicating that zoom is enabled or whether it is undepressed indicatingthat zoom is disabled.

The display device 102 may be, and in some embodiments is, a touchscreen, used to display images, video, information regarding theconfiguration of the camera device, and/or status of data processingbeing performed on the camera device. In the case where the displaydevice 102 is a touch screen, the display device 102 serves as anadditional input device and/or as an alternative to the separate inputdevice, e.g., buttons, 106. As will be discussed in some embodimentszooming operation can be controlled by pressing a zoom control sensor,e.g., a touch sensor. In some embodiments when the camera user touchesthe zoom control sensor the zoom functionality is enabled. For example afinger on the touch sensor activates/enables the zoom functionality. TheI/O interface 112 couples the display 102 and input device 106 to thebus 116 and interfaces between the display 102, input device 106 and theother elements of the camera which can communicate and interact via thebus 116.

In addition to being coupled to the I/O interface 112, the bus 116 iscoupled to the memory 108, processor 110, an optional autofocuscontroller 132, the wireless and/or wired interface 114, a zoom controlmodule 140, and a plurality of optical chains 130, e.g., X opticalchains also referred to herein as camera modules. In some embodiments Xis an integer greater than 2, e.g., 3, 4, 7 or a larger value dependingon the particular embodiment. The plurality of camera modules 130 may beimplemented using any of the various camera module sets and/orarrangements described in the present application. For example, in someembodiments the camera device 100 is implemented using a set of cameramodules as shown in FIG. 12A while in other embodiments the cameradevice 100 is implemented using the module arrangement shown in FIG. 13Bor FIG. 17A or any one of the other Figures included in thisapplication. Images captured by individual optical chains in theplurality of optical chains 130 can, and in various embodiments are,stored in memory 108, e.g., as part of the data/information 120 andprocessed by the processor 110, e.g., to generate one or more compositeimages.

The X camera modules 131 through 133 may, and in various embodiments do,include camera modules having different focal lengths. Multiple cameramodules may be provided at a given focal length. For example, multiplecamera modules having a 35 mm equivalent focal length to a full frameDSLR camera, multiple camera modules having a 70 mm equivalent focallength to a full frame DSLR camera and multiple camera modules having a140 mm equivalent focal length to a full frame DSLR camera are includedin an individual camera device in some embodiments. The various focallengths are exemplary and a wide variety of camera modules withdifferent focal lengths may be used. The camera device 100 is to beconsidered exemplary. To the extent that other references are made to acamera or camera device with regard to some of the other figures, it isto be understood that at least in some embodiments the camera device orcamera will include the elements shown in FIG. 1 even if the elementsare not shown in a particular figure or embodiment. While in someembodiments all of the elements shown in FIG. 1 are included in thecamera device or camera, in other embodiments a subset of the elementsshown in FIG. 1 are included and the illustration of the elements inFIG. 1 is not intended to imply that a particular element is essentialor necessary in all embodiments.

As will be discussed below images from different camera modules capturedat the same time or during a given time period can be combined togenerate a composite image, e.g., an image having better resolution,frequency content and/or light range than an individual image capturedby a single one of the camera modules 131, 133.

Multiple captured images and/or composite images may, and in someembodiments are, processed to form video, e.g., a series of imagescorresponding to a period of time. The interface 114 couples theinternal components of the camera device 100 to an external network,e.g., the Internet, and/or one or more other devices e.g., memory orstand alone computer. Via interface 114 the camera device 100 can anddoes output data, e.g., captured images, generated composite images,and/or generated video. The output may be to a network or to anotherexternal device for processing, storage and/or to be shared. Thecaptured image data, generated composite images and/or video can beprovided as input data to another device for further processing and/orsent for storage, e.g., in external memory, an external device or in anetwork.

The interface 114 of the camera device 100 may be, and in some instancesis, coupled to a computer so that image data may be processed on theexternal computer. In some embodiments the external computer has ahigher computational processing capability than the camera device 100which allows for more computationally complex image processing of theimage data outputted to occur on the external computer. The interface114 also allows data, information and instructions to be supplied to thecamera device 100 from one or more networks and/or other externaldevices such as a computer or memory for storage and/or processing onthe camera device 100. For example, background images may be supplied tothe camera device to be combined by the camera processor 110 with one ormore images captured by the camera device 100. Instructions and/or dataupdates can be loaded onto the camera via interface 114 and stored inmemory 108.

The lighting module 104 in some embodiments includes a plurality oflight emitting elements, e.g., LEDs, which can be illuminated in acontrolled manner to serve as the camera flash with the LEDs beingcontrolled in groups or individually, e.g., in a synchronized mannerbased on operation of the rolling shutter and/or the exposure time. Forpurposes of discussion module 104 will be referred to as an LED modulesince in the exemplary embodiment LEDs are used as the light emittingdevices but as discussed above the invention is not limited to LEDembodiments and other light emitting sources may be used as well. Insome embodiments the LED module 104 includes an array of light emittingelements, e.g., LEDs. In some embodiments the light emitting elements inthe LED module 104 are arranged such that each individual LED and/or agroup of LEDs can be illuminated in a synchronized manner with rollingshutter operation. Light emitting elements are illuminated, in some butnot all embodiments, sequentially, so that different portions of an areaare illuminated at different times so that the full area need not beconsistently lighted during image capture. While all lighting elementsare not kept on for the full duration of an image capture operationinvolving the reading out of the full set of pixel elements of a sensor,the portion of area which is having its image captured, e.g., the scanarea, at a given time as a result of the use of a rolling shutter willbe illuminated thanks to synchronization of the lighting of lightemitting elements with rolling shutter operation. Thus, various lightemitting elements are controlled to illuminate at different times insome embodiments based on the exposure time and which portion of asensor will be used to capture a portion of an image at a given time. Insome embodiments the light emitting elements in the LED module 104include a plurality of sets of light emitting elements, each set oflight emitting elements corresponding to a different image area which itilluminates and which is captured by a different portion of the imagesensor. Lenses may, and in some embodiments are used to direct the lightfrom different light emitting elements to different scene areas whichwill be captured by the camera through the use of one or more cameramodules.

The rolling shutter controller 150 is an electronic shutter thatcontrols reading out of different portions of one or more image sensorsat different times. Each image sensor is read one row of pixel values ata time and the various rows are read in order. As will be discussedbelow, the reading out of images captured by different sensors iscontrolled in some embodiments so that the sensors capture a scene areaof interest, also sometimes referred to as an image area of interest, ina synchronized manner with multiple sensors capturing the same imagearea at the same time in some embodiments.

While an electronic rolling shutter is used in most of the embodiments,a mechanical rolling shutter may be used in some embodiments.

The light control device 152 is configured to control light emittingelements (e.g., included in the LED module 104) in a synchronized mannerwith the operation of the rolling shutter controller 150. In someembodiments the light control device 152 is configured to controldifferent sets of light emitting elements in the array to emit light atdifferent times in a manner that is synchronized with the timing of therolling shutter 150. In some embodiments the light control device 152 isconfigured to control a first set of light emitting elementscorresponding to a first image area to output light during a first timeperiod, the first time period being determined based on the timing ofthe rolling shutter and being a period of time during which a firstportion of the sensor is exposed for image capture. In some embodimentsthe light control device 152 is further configured to control a secondset of light emitting elements corresponding to a second image area tooutput light during a second time period, the second time period beingdetermined based on the timing of the rolling shutter and being a periodof time during which a second portion of the sensor is exposed for imagecapture. In some embodiments the first time period includes at least aportion of time which does not overlap the second time period.

In some embodiments the light control device 152 is further configuredto control an Nth set of light emitting elements corresponding to an Nthimage area to output light during a third time period, said Nth timeperiod being determined based on the timing of the rolling shutter andbeing a period of time during which an Nth portion of the sensor isexposed for image capture, N being an integer value corresponding to thetotal number of time periods used by said rolling shutter to completeone full read out of total image area.

In some embodiments the light control device 152 is further configuredto control the second set of light emitting elements to be off duringsaid portion of time included in the first period of time which does notoverlap said second period of time. In some embodiments the lightcontrol device is configured to determine when the first set and saidsecond set of light emitting elements are to be on based on an exposuresetting. In some embodiments the light control device is configured todetermine when said first set and said second set of light emittingelements are to be on based on an amount of time between read outs ofdifferent portions of said sensor. In some embodiments the differentsets of light emitting elements in the plurality of light emittingelements are covered with different lenses. In some such embodiments thelight control device 152 is further configured to determine which setsof light emitting elements to use based on an effective focal lengthsetting being used by the camera device.

The accelerometer module 122 includes a plurality of accelerometersincluding accelerometer 1 124, accelerometer 2 126, and accelerometer 3128. Each of the accelerometers is configured to detect cameraacceleration in a given direction. Although three accelerometers 124,126 and 128 are shown included in the accelerometer module 122 it shouldbe appreciated that in some embodiments more than three accelerometerscan be used. Similarly the gyro module 192 includes 3 gyros, 194, 196and 198, one for each axis which is well suited for use in the 3dimensional real world environments in which camera devices are normallyused. The camera acceleration detected by an accelerometer in a givendirection is monitored. Acceleration and/or changes in acceleration, androtation indicative of camera motion, are monitored and processed todetect one or more directions, of motion e.g., forward camera motion,backward camera motion, etc. As discussed below, theacceleration/rotation indicative of camera motion can be used to controlzoom operations and/or be provided in some cases to a camera mount whichcan then take actions such as rotating a camera mount or rotating acamera support to help stabilize the camera.

The camera device 100 may include, and in some embodiments does include,an autofocus controller 132 and/or autofocus drive assembly 134. Theautofocus drive assembly 134 is, in some embodiments, implemented as alens drive. The autofocus controller 132 is present in at least someautofocus embodiments but would be omitted in fixed focus embodiments.The autofocus controller 132 controls adjustment of at least one lensposition in one or more optical chains used to achieve a desired, e.g.,user indicated, focus. In the case where individual drive assemblies areincluded in each optical chain, the autofocus controller 132 may drivethe autofocus drive of various optical chains to focus on the sametarget.

The zoom control module 140 is configured to perform a zoom operation inresponse to user input.

The processor 110 controls operation of the camera device 100 to controlthe elements of the camera device 100 to implement the steps of themethods described herein. The processor may be a dedicated processorthat is preconfigured to implement the methods. However, in manyembodiments the processor 110 operates under direction of softwaremodules and/or routines stored in the memory 108 which includeinstructions that, when executed, cause the processor to control thecamera device 100 to implement one, more or all of the methods describedherein. Memory 108 includes an assembly of modules 118 wherein one ormore modules include one or more software routines, e.g., machineexecutable instructions, for implementing the image capture and/or imagedata processing methods of the present invention. Individual stepsand/or lines of code in the modules of 118 when executed by theprocessor 110 control the processor 110 to perform steps of the methodof the invention. When executed by processor 110, the data processingmodules 118 cause at least some data to be processed by the processor110 in accordance with the method of the present invention. The assemblyof modules 118 includes a mode control module which determines, e.g.,based on user input which of a plurality of camera device modes ofoperation are to be implemented. In different modes of operation,different camera modules 131, 133 may and often are controlleddifferently based on the selected mode of operation. For example,depending on the mode of operation different camera modules may usedifferent exposure times. Alternatively, the scene area to which thecamera module is directed and thus what portion of a scene is capturedby an individual camera module may be changed as will be discussed belowwith regard to FIGS. 5 and 34 depending on how the images captured bydifferent camera modules are to be used, e.g., combined to form acomposite image and what portions of a larger scene individual cameramodules are to capture during the user selected or automaticallyselected mode of operation. In some embodiments, the operationsperformed by the processor when executing the instructions from one ormore assembly of modules is instead performed by a hardware module whichperforms the same functionality and is included in the hardware assemblyof modules.

The resulting data and information (e.g., captured images of a scene,combined images of a scene, etc.) are stored in data memory 120 forfuture use, additional processing, and/or output, e.g., to displaydevice 102 for display or to another device for transmission, processingand/or display. The memory 108 includes different types of memory forexample, Random Access Memory (RAM) in which the assembly of modules 118and data/information 120 may be, and in some embodiments are stored forfuture use. Read only Memory (ROM) in which the assembly of modules 118may be stored for power failures. Non-volatile memory such as flashmemory for storage of data, information and instructions may also beused to implement memory 108. Memory cards may be added to the device toprovide additional memory for storing data (e.g., images and video)and/or instructions such as programming. Accordingly, memory 108 may beimplemented using any of a wide variety of non-transitory computer ormachine readable mediums which serve as storage devices.

Having described the general components of the camera device 100 withreference to FIG. 1, various features relating to the plurality ofoptical chains 130 will now be discussed with reference to FIGS. 2 and 3which show the camera device 100 from front and side perspectives,respectively. Dashed line 101 of FIG. 2 indicates a cross section line.

Box 117 represents a key and indicates that OC=optical chain, e.g.,camera module, and each L1 represents an outermost lens in an opticalchain. Box 119 represents a key and indicates that S=sensor, F=filter,L=lens, L1 represents an outermost lens in an optical chain, and L2represents an inner lens in an optical chain. While FIG. 3 shows onepossible implementation of optical chains, as will be discussed below,other embodiments are possible and the optical chains may include one ormore light redirection elements in addition to the elements shown inFIG. 3. The lenses of different optical chains may have differentshapes, e.g., with round apertures being used for some lenses andnon-round apertures being used for other lenses. However, in someembodiments lenses with round apertures are used for each of the opticalchains of a camera device.

FIG. 2 shows the front of the exemplary camera device 100. Rays of light131, which is light toward the front of the camera assembly, shown inFIG. 1 may enter the lenses located in the front of the camera housing.From the front of camera device 100, the camera device 100 appears as arelatively flat device with the outer rectangle representing the camerahousing and the square towards the center of the camera representing theportion of the front camera body in which the plurality of opticalchains 130 is mounted. Note that while outer lenses shown in FIG. 2 areshown as having circular apertures which are the same size, as will bediscussed below different size lenses may be used for different opticalchains, e.g., depending on the focal length with optical chains havinglarger focal lengths normally including outer lenses with largerapertures than optical chains with small focal lengths.

FIG. 3, which shows a side perspective of camera device 100, illustratesthree of the seven optical chains (OC 1 121, OC 7 145, OC 4 133) of theset of optical chains 130, display 102 and processor 110. OC 1 121includes an outer lens L1 103, a filter 123, an inner lens L2 125, and asensor 127. In some embodiments the OC 1 121 further includes lens drive(LD) 129 for controlling the position of lens L2 125 for zooming and/orauto focus operation purposes. The exposure and read out controller 150is not shown in the figure but is used for controlling the read out ofrows of pixel values form the sensors' 127, 151 and 139 in asynchronized manner, e.g., taking into consideration the scene areabeing captured by the individual sensors. The LD 129 includes a motor orother drive mechanism which can move the lens, barrel or cylinderhousing one or more lenses, or sensor, to which it is connected therebyallowing for an alteration to the light path by moving one or moreelements relative to the other elements of the optical chain to whichthe LD is coupled. While the LD 129 is shown coupled, e.g., connected,to the lens L2 125 and thus can move the position of the lens L2, e.g.,as part of a zooming or autofocus operation, in other embodiments the LD129 is coupled to a cylindrical or barrel shape component which is partof the optical chain or to the sensor 127. Thus, the lens drive canalter the relative position of a lens to the sensor 127, e.g., to changethe distance between the sensor 127 and the lens 125 as part of azooming and/or focus operation. OC 7 145 includes an outer lens L1 115,a filter 147, an inner lens L2 149, and a sensor 151. OC 7 145 furtherincludes LD 153 for controlling the position of lens L2 149. The LD 153includes a motor or other drive mechanism which can move the lens,barrel, cylinder, sensor or other optical chain element to which it isconnected.

OC 4 133 includes an outer lens L1 109, a filter 135, an inner lens L2137, and a sensor 139. OC 4 133 includes LD 141 for controlling theposition of lens L2 137. The LD 141 includes a motor or other drivemechanism and operates in the same or similar manner as the drives ofthe other optical chains. While only three of the OCs are shown in FIG.3 it should be appreciated that the other OCs of the camera device 100may, and in some embodiments do, have the same or similar structureand/or may include other elements such as light redirection devices.Thus, differences between the multiple optical chains of the cameradevice 100 are possible and, in some embodiments, are present to allowfor a variety of focal lengths to be supported in a single camera devicethrough the use of multiple optical chains which can be operated inparallel.

FIG. 3 and the optical chains (OCs), also sometimes referred to ascamera modules, illustrated therein are illustrative of the generalstructure of OCs used in various embodiments. However, numerousmodifications and particular configurations are possible. Whilereference to elements of FIG. 3 may be made, it is to be understood thatthe OCs (camera modules) in a particular embodiment will be configuredas described with regard to the particular embodiment and that variousdifferent camera modules are often used in single camera device. FIG. 5and FIG. 13A show optical chains, e.g., camera modules, which includelight redirection devices. Such modules can be used alone or incombination with other modules such as the ones shown in FIGS. 3 and 4Aor other figures of the present application.

While a filter may be of a particular color or used in some opticalchains, filters need not be used in all optical chains and may not beused in some embodiments. In embodiments where the filter is expresslyomitted and/or described as being omitted or an element which allows alllight to pass, while reference may be made to the OCs of FIG. 3 itshould be appreciated that the filter will be omitted in an embodimentwhere it is indicated to be omitted or of such a nature that it allows abroad spectrum of light to pass if the embodiment is indicated to have abroadband filter. While in the OCs of FIG. 3 light redirection devices(R), e.g., mirrors or prisms are not shown, as will be discussed below,in at least some embodiments one or more mirrors are included in OCs forlight to be redirected, e.g., to increase the length of the optical pathor make for a more convenient internal component configuration. Itshould be appreciated that each of the OCs 121, 145, 133, shown in FIG.3 will have their own optical axis. In the example, each optical axispasses through the center of the lens 103, 115, or 109 at the front ofthe optical chain and passes through the OC to the corresponding sensor127, 151, 139.

While the processor 110 is not shown being coupled to the LD, andsensors 127, 151, 139 it is to be appreciated that such connectionsexist and are omitted from FIG. 3 to facilitate the illustration of theconfiguration of the exemplary OCs.

As should be appreciated the number and arrangement of lens, filtersand/or mirrors can vary depending on the particular embodiment and thearrangement shown in FIG. 3 is intended to be exemplary and tofacilitate an understanding of various features rather than to belimiting in nature.

The front of the plurality of optical chains 130 is visible in FIG. 2with the outermost lens of each optical chain appearing as a circlerepresented using a solid line (OC 1 L1 103, OC 2 L1 105, OC 3 L1 107,OC 4 L1 109, OC 5 L1 111, OC 6 L1 113, OC 7 L1 115). In the FIG. 2example, the plurality of optical chains 130 include seven opticalchains, OC 1 121, OC 2 157, OC 3 159, OC 4 133, OC 5 171, OC 6 173, OC 7145, which include lenses (OC 1 L1 103, OC 2 L1 105, OC 3 L1 107, OC 4L1 109, OC 5 L1 111, OC 6 L1 113, OC 7 L1 115), respectively,represented by the solid circles shown in FIG. 2. The lenses of theoptical chains are arranged to form a pattern which is generallycircular in the FIG. 2 example when viewed as a unit from the front.While a circular arrangement is used in some embodiments, non-circulararrangements are used and preferred in other embodiments. In someembodiments while the overall pattern is generally or roughly circular,different distances to the center of the general circle and/or differentdistances from one lens to another is intentionally used to facilitategeneration of a depth map and block processing of images which mayinclude periodic structures such as repeating patterns without the needto identify edges of the repeating pattern. Such repeating patterns maybe found in a grill or a screen.

The overall total light capture area corresponding to the multiplelenses of the plurality of optical chains OC 1 to OC 7, also sometimesreferred to as optical camera modules, can, in combination, approximatethat of a lens having a much larger opening but without requiring asingle lens having the thickness which would normally be necessitated bythe curvature of a single lens occupying the area which the lenses shownin FIG. 2 occupy.

While gaps are shown between the lens openings of the optical chains OC1 to OC 7, it should be appreciated that the lenses may be made, and insome embodiments are, made so that they closely fit together minimizinggaps between the lenses represented by the circles formed by solidlines. While seven optical chains are shown in FIG. 2, it should beappreciated that other numbers of optical chains are possible. Forexample, as shown in FIGS. 12A and 17A seventeen camera modules are usedin a single camera device in some embodiments. Camera devices includingeven larger numbers of optical chains are also possible.

The use of multiple optical chains has several advantages over the useof a single optical chain. Using multiple optical chains allows fornoise averaging. For example, given the small sensor size there is arandom probability that one optical chain may detect a different number,e.g., one or more, photons than another optical chain. This mayrepresent noise as opposed to actual human perceivable variations in theimage being sensed. By averaging the sensed pixel values correspondingto a portion of an image, sensed by different optical chains, the randomnoise may be averaged resulting in a more accurate and pleasingrepresentation of an image or scene than if the output of a singleoptical chain was used.

Given the small size of the optical sensors (e.g., individual pixelelements) the dynamic range, in terms of light sensitivity, is normallylimited with the sensors becoming easily saturated under brightconditions. By using multiple optical chains corresponding to differentexposure times the dark portions of a scene area can be sensed by thesensor corresponding to the longer exposure time while the lightportions of a scene area can be sensed by the optical chain with theshorter exposure time without getting saturated. Pixel sensors of theoptical chains that become saturated as indicated by a pixel valueindicative of sensor saturation can be ignored, and the pixel value fromthe other, e.g., less exposed, optical chain can be used withoutcontribution from the saturated pixel sensor of the other optical chain.Weighting and combining of non-saturated pixel values as a function ofexposure time is used in some embodiments. By combining the output ofsensors with different exposure times a greater dynamic range can becovered than would be possible using a single sensor and exposure time.

FIG. 3 is a cross section perspective of the camera device 100 shown inFIGS. 1 and 2. Dashed line 101 in FIG. 2 shows the location within thecamera device to which the cross section of FIG. 3 corresponds. From theside cross section, the components of the first, seventh and fourthoptical chains are visible.

As illustrated in FIG. 3 despite including multiple optical chains thecamera device 100 can be implemented as a relatively thin device, e.g.,a device less than 2, 3 or 4 centimeters in thickness in at least someembodiments. Thicker devices are also possible, for example devices withtelephoto lenses, and are within the scope of the invention, but thethinner versions are particularly well suited for cell phones and/ortablet implementations. As will be discussed below, various techniquessuch as the use of light redirection elements and/or non-circular lensescan be used in conjunction with small sensors, such as those commonlyused in handheld cameras, to support relatively large focal lengths,e.g., camera modules of 150 mm equivalent focal length to a full frameDSLR camera, 300 mm equivalent focal length to a full frame DSLR cameraor above in a relatively thin camera device format.

As illustrated in the FIG. 3 diagram, the display device 102 may beplaced behind the plurality of optical chains 130 with the processor110, memory and other components being positioned, at least in someembodiments, above or below the display and/or optical chains 130. Aswill be discussed below, and as shown in FIG. 3, each of the opticalchains OC 1 121, OC 7 145, OC 4 133 may, and in some embodiments do,include an outer lens L1, an optional filter F, and a second optionallens L2 which proceed a sensor S which captures and measures theintensity of light which passes through the lens L1, filter F and secondlens L2 to reach the sensor S. The filter may be a color filter or oneof a variety of other types of light filters or may be omitted dependingon the particular optical chain embodiment or configuration.

Note that while supporting a relatively large light capture area andoffering a large amount of flexibility in terms of color filtering andexposure time, the camera device 100 shown in FIG. 3 is relatively thinwith a thickness that is much less, e.g., ⅕th, 1/10th, 1/20th or evenless than the overall side to side length or even top to bottom lengthof the camera device visible in FIG. 2.

FIG. 4A illustrates a camera device 200 implemented in accordance withthe invention. The FIG. 4 camera device 200 includes many or all of thesame elements shown in the device 100 of FIGS. 1-3. Exemplary cameradevice 200 includes a plurality of optical chains (OC 1 205, OC 2 207, .. . , OC X 209, a processor 211, memory 213 and a display 215, coupledtogether. OC 1 205 includes outer lens L1 251, a light redirectionelement R 252, a hinge (or mirror) drive HD 291, filter 253, inner lensL2 255, sensor 1 257, and LD 259. The HD 291 can be used to move aposition of a hinge to which the light redirection device (R) 252, e.g.,mirror, is mounted and thus move the mirror to change the scene area towhich the module 205 is directed without moving the lens 251. Moving(e.g., rotating about a hinge) the mirror 252 to change the scene areato which the module 205 is directed is especially useful in anembodiment where the outer lens 251 is a plane piece of glass or aplastic piece with no optical power as is the case in some embodiments.

The optical chains shown in FIG. 4A can be arranged in various positionswithin the camera 200. The elements in FIG. 4B which are the same asthose shown in FIG. 4A are identified using the same references numbersand will not be described again. FIG. 4B shows the configuration of theoptical chains in an arrangement where light enters via the front orface of the camera 200 and is redirected to sensors 257, 269, 281, ofthe first through third camera modules respectively, mounted on theinside top portion of the camera housing which forms the outer portionof camera 200.

As can be seen in the FIG. 4B embodiment, light entering in thehorizontal dimension is redirected upward in the vertical. For example,light entering through outer lens 251 of the first optical chain 205 isredirected upward by mirror 252 so that it passes though filter 253 andinner lens 255 as it travels towards sensor 257. An optical chain suchas the first optical chain 205, that has a light redirection element,such as the element 252, can be divided, for purposes of discussion,into two parts, Part A and Part B. Part A consists of all those elementsin the optical chain that are in the light path before the lightredirection element 252 and Part B consists of all the optical elements(including the image sensor) that are in the light path after the lightredirection element. The optical axis of the optical chain 205 as seenfrom outside the camera is the optical axis 291 of Part A. Lighttraveling into the optical chain 205 along the optical axis 291 will beredirected upward along the optical axis 293 of Part B of the firstoptical chain.

In one particular exemplary embodiment of the optical chain 205, Part Acontains no optical elements with any optical power, e.g., Part Acontains plane glass or filters but no lenses. In this case the opticalaxis of the optical chain as seen from outside the camera is simplyalong a light path that gets redirected along the optical axis 293 ofPart B by the light redirection element. In some embodiments one or morelenses 255 are included in Part B of the optical chain which have anoptical power. Thus, it should be appreciated that in at least someembodiments the outer lens 251 may be implemented as a flat orrelatively flat lens which does not protrude from the surface of thecamera 200. This reduces the risk of scratches and also reduces thepossibly that the outer lens will get caught when inserting or removingit from a pocket or case as might be the case if the lens protruded fromthe camera.

It should be appreciated that the optical axis of the second and thirdcamera modules are similar to that of the first optical module 205 andthat the components of the optical chains may also be grouped into twoparts, Part A which corresponds to components proceeding the mirror ofthe optical chain and Part B which corresponds to components subsequentthe mirror of the optical chain. From the perspective of the opticalpath of an optical chain, the optical path like the components may begrouped as Part A and Part B with the mirror providing the transitionpoint between Part A of an optical path and Part B of the optical path.

In some but not all embodiments, processor 211 of camera device 200 ofFIG. 4A is the same as or similar to processor 110 of device 100 of FIG.1, memory 213 of device 200 of FIG. 4A is the same as or similar to thememory 108 of device 100 of FIG. 1, the zoom control module 214 ofdevice 200 is the same as or similar to the zoom control module 140 ofdevice 100, the accelerometer module 216 of device 200 is the same as orsimilar to the accelerometer module 122 of device 100 and display 215 ofdevice 200 of FIG. 4A is the same as or similar to the display 102 ofdevice 100 of FIG. 1.

OC 2 207 includes outer lens L1 263, light redirection device 231, hingedrive 293, filter 265, inner lens L2 267, sensor 2 269, and LD 271. OC N209 includes outer lens L1 275, light redirection device 235, hingedrive 295, filter 277, inner lens L2 279, sensor N 281, and LD 283. Theexposure and read out controller 150 controls sensors to read out, e.g.,rows of pixel values, in a synchronized manner while also controllingthe exposure time. In some embodiments the exposure and read outcontroller 150 is a rolling shutter controller including an exposurecontroller 287 and a sensor read out controller 289. An autofocuscontroller 152 is included to control the lens drives 259, 271 and 283in some embodiments.

In the FIG. 4A embodiment the optical chains (optical chain 1 205,optical chain 2 207, . . . , optical chain N 209) are shown asindependent assemblies with the lens drive of each module being aseparate LD element (LD 259, LD 271, LD 283), respectively. Each of theLDs shown adjusts the position of the corresponding lens to which it isconnected as part of a zooming and/or focus operation. In someembodiments the LD controls the position of a lens and/or sensor inwhich case the LD is connected to both a lens support mechanism or lensand the sensor.

In FIG. 4A, the structural relationship between the mirror and variouslenses and filters which precede the sensor in each optical chain can beseen more clearly than in some of the other figures. While fourelements, e.g. two lenses (see columns 201 and 203 corresponding to L1and L2, respectively), a light redirection device R (see col. 217), andthe filter (corresponding to column 202) are shown in FIG. 4A beforeeach sensor, it should be appreciated that a much larger combinations(e.g., numbers) of lenses, light redirection elements and/or filters mayprecede the sensor of one or more optical chains with anywhere from 2-10elements being common and an even larger number of elements being usedin some embodiments, e.g., high end embodiments and/or embodimentssupporting a large number of filter and/or lens options. Furthermore itshould be appreciated that all illustrated elements need not be includedin all optical chains. For example, in some embodiments optical chainshaving relatively short focal lengths may be implemented without the useof a light redirection element being used, e.g., to redirect the lightby 90 degrees, since the optical chain with a short focal length can beimplemented in a straight but still relatively compact manner given theshort focal length.

In some but not all embodiments, optical chains are mounted in thecamera device with some, e.g., the shorter focal length optical chainsextending in a straight manner from the front of the camera devicetowards the back. However, in the same camera, longer focal lengthcamera modules may and sometimes do include light redirection deviceswhich allow at least a portion of the optical path of a camera module toextend sideways allowing the length of the optical axis to be longerthan the camera is deep. The use of light redirection elements, e.g.,mirrors, is particularly advantageous for long focal length cameramodules given that the overall length of such modules tends to be longerthan that of camera modules having shorter focal lengths. A camera mayhave a wide variety of different camera modules some with lightredirection elements, e.g., mirrors, and others without mirrors. Filtersand/or lenses corresponding to different optical chains may, and in someembodiments are, arranged in planes, e.g. the apertures of the outermostlenses may be configured in a plane that extends parallel to the face ofthe camera, e.g., a plane in which the front of the camera both extendsvertically and horizontally when the camera is in a vertical directionwith the top of the camera both being up.

FIG. 5 shows an optical chain, e.g., camera module, 500 which is used invarious exemplary embodiments. A plurality of optical chains of the typeillustrated in FIG. 5 are used in a camera device such as camera 600discussed in detail below. The camera module 500 is an optical chainwhich includes an outer lens 512, a light redirection device, e.g.,mirror, 510 positioned behind the lens 512, a hinge drive 516, a mirrorhinge 508, a first cylindrical module portion 506, a second cylindricalmodule portion 504, a sensor 502 and a lens drive 514. Light enters theoptical chain 500 via the lens 512 and is redirected by the mirror 510so that it reaches the sensor 502 at the back of the optical chain. Thefirst and second cylindrical portions 504, 506 can house one or morelenses or filters as well as other optical components through whichlight may pass before reaching the sensor 502. While the mirror 510 isnormally used to redirect light 90 degrees so that light enteringthrough the lens 512 (which may be positioned on the face of the camera)along it's optical axis will be redirected along the optical axis ofPart B of the optical chain 500 so that is travels towards the side ofthe camera allowing for the optical chain 500 to effectively use theside to side distance of the camera device in which the optical chain500 is mounted, the hinge drive 516 may move the position of the hinge508 and thus the mirror 510 to alter the angle of redirection so that itvaries from 90 degrees. Thus, the direction in which the optical chain500 effectively points may be altered by moving all or a portion of thehinge 508 and mirror 510 without moving the lens 512. In someembodiments, the axis of the hinge is perpendicular to the Part B of theoptical axis and parallel to the place of the front face of the camera600. In some embodiments, the lens 512 is plane glass with no opticalpower.

The hinge drive may be implemented using a motor or other mechanicalmechanisms which can be used to drive or change the position of themirror 510 and/or hinge 508 which connects the mirror to the othercomponents of the camera module such as cylindrical portion 506.

The cylindrical or barrel portions 504, 506 may be moved by drive 514 sothat they slide relative to each other, e.g., barrel portion 504 may bemoved so that it moves further into or out of the barrel portion 506thereby altering the distance from the lens 512 to the sensor 502 aspart of a focus or zoom operation.

It should be appreciated that the optical chain 500 allows forrelatively long optical chains to be positioned in a camera device whichhas a depth which is less than the overall length of the optical chain500. The camera module 500 is particular well suited for implementingcamera devices which include multiple optical chains but which are stillintended to be relatively thin to facilitate storage in a pocket orother storage device.

FIG. 6 is a perspective view of a camera device 600 which includes aplurality of the optical chains, e.g., camera modules of the type shownin FIG. 5. The camera 600 includes a cover 602 which is flexible and canbe slid to the side to form a hand grip and to expose the lens area 604.In the FIG. 6 embodiment, the lens area 604 includes a plurality ofouter lenses each represented by a circle. Larger circles correspond tooptical chains with larger apertures and focal lengths than the opticalchains with smaller lenses. The FIG. 6 embodiment includes a total of 17optical chains corresponding to three different focal lengths. There are5 of the small focal length optical chains as can be seen by the 5smallest circles in the lens area 604, 5 medium focal length opticalchains as can be seen by the 5 medium sized circles representing theouter lenses of the 5 medium focal length optical chains and 7 longfocal length optical chains as can be seen by the seven larger circlesshown in the FIG. 6 lens area 604. The focal length relationship betweenthe smallest and largest optical chains, in one embodiment is such thatthe smallest focal length is ¼ the focal length of the largest opticalchain and ½ the focal length of the medium focal length optical chains.For example, the small, medium and large focal length optical chainsmay, and in one embodiment do, have equivalent full frame DSLR focallengths of 35 mm, 70 mm and 140 mm respectively. It should beappreciated that such a difference in focal lengths will result in the35 mm camera module capturing a scene area approximately four timeslarger than the area captured by the 70 mm camera module and 16 timesthe size of the scene area captured by the camera module having the 140mm focal length. While not shown in FIG. 6, it should be appreciatedthat camera device 600 may and in some embodiments does include the sameor similar elements as camera device 100 and camera device 200 of FIGS.1 and 4A. Thus it should be appreciated that camera device 600 includesvarious elements such as the processor 110/211, memory 108/213, zoomcontroller 140/214, exposure and read out controller 150, accelerometer,gyro, autofocus controller 132 etc., and various other elementsdiscussed above with regard to camera devices 100 and 200.

By using camera modules having different focal lengths to captureportion of a scene area of interest, and by then combining the images asdone in various embodiments, the composite image can have a higheroverall pixel count than any one individual sensor. Accordingly, evenwhen the sensors used by different camera modules having different focallengths have the same pixel count, the number of pixels of the compositeimage can be far higher than the number of pixels of an individualsensor.

In some embodiments, as will be discussed below, different portions of ascene area of interest are captured by different ones of the cameramodules having the largest focal length. The camera modules with mediumor small focal lengths are then used to capture larger portions of ascene area of interest where the larger scene area of interest maycorrespond to a complete scene area of interest, e.g., in the case ofthe camera modules of the camera device using the smallest supportedfocal length. Overlapping images are captured by image sensors ofdifferent camera modules. Based on known spatial information aboutposition of the camera modules on the camera device, e.g., the distancebetween the outer lenses of the different camera modules and/or theangle at which the each of the individual camera modules (which capturean image to be combined) points, depth information is generated. Usingthe depth information images captured by different camera modules arecombined to form a composite image as will be discussed further below.Note that the images can, and in some embodiments are, combined withoutgenerating depth information in a scene or scene area. In some suchembodiments an image covering the scene area of interest is first chosenas a reference image. The composite image is then generated from theperspective captured in the reference image. In one such embodiment, achosen small block of pixels in the reference image is combined with amatching collection of pixels in each image included in a chosen subsetof other images. The combining is performed such that each pixel in thereference-image-block is combined with the matching pixel (orinterpolated pixel if the image portions match with the non-integerpixel shift) in each of the images in the subject. The combination canbe a weighted sum of pixels values, where the weights are assigned basedon the quality of the image to which the pixel belongs. This combinedblock of pixels now becomes a corresponding block of the compositeimage. The process is repeated for the other blocks of the referenceimage to generate the entire composite image. Note that images takenwith the camera modules with different focal lengths then the referenceimage have different magnifications. In this case these images shouldfirst be appropriately scaled to have the same magnification as thereference image before above process is carried out. The composite imagegenerated by combining multiple images as above as likely to have betterSNR then the reference image which was selected of the basis of theimage combining used to generate the composite image.

The camera device of FIG. 6 with its 17 different camera modulescorresponding to three different focal lengths is particularly wellsuited for combining captured images to generate a composite image aswill be discussed in detail below.

FIG. 7 illustrates the camera device 600 of FIG. 6 with the camera casein the closed position. Note that the flexible case 602 serves as acover which can protect the lens area 604 and the lenses includedtherein when the camera is not in use.

FIG. 8 is a frontal view of the camera device 600 and the lensarrangement of the camera device with the 15 outer lenses being clearlyvisible as circles in the lens area 604. Note that the diameter of thesmallest lenses is d which correspond to the camera modules having thesmallest focal length, the outer lenses corresponding to the mediumfocal length modules have a diameter 2d, and the camera modules havingthe largest focal length have a diameter 4d. This results in the cameramodules having the same ‘f stop’ or ‘f number’ given the focal lengthrelationship f1 being ¼ the largest focal length (f3) and one half thefocal length of the medium focal length f2 of the camera modules havinga medium focal length. The ‘f number’ is the ratio of the focal lengthto the aperture diameter and determines the diffraction limit of thecamera modules. The smaller the f number, the less likely it is that thecamera module will be diffraction limited. Smaller f numbers usuallycorresponded to larger optical complexity in the camera module. Smalllenses with 5 or 6 molded plastic elements these days can bemanufactured in a cost effective manner for f numbers around 2.4.Accordingly, in some embodiments plastic lenses made of multiple plasticelements are used.

FIG. 9 is a simplified illustration of the camera device 600 with thecase, controls and other features being omitted to allow a betterappreciation of the lens configuration and lens area.

FIG. 10 is a side view 1000 of the camera device 600. As shown in FIG.10, the camera device has a height H (not including control buttons) anda depth D1. D1 is equal to or greater than the diameter D of theaperture of the largest lens shown in FIG. 6. As will be discussedbelow, depending on the embodiment, the diameter of outer lenses withround apertures can have an impact on the minimum depth of the camera inthe case where light redirection elements, e.g., mirrors, are used toredirect light. In this case the minimum depth of the camera must begreater than this diameter. If this is not the case the actual moduleaperture will be smaller than the diameter of the outer lens and asmaller outer lens could have been used.

FIG. 11 shows an enlarged version 1100 of the outer lens arrangement ofthe camera 600. In FIG. 11 the outer lenses of the three different sizescan be clearly seen with the largest diameter lenses corresponding tocamera modules having the largest focal length and thus zoom, e.g.,magnification.

FIG. 12A is a diagram 1200 showing how the 17 optical chains, e.g.,camera modules, of the camera 600 can be arranged within the body of thecamera 600. The seven optical chains 1202, 1206, 1210, 1212, 1216 1220,1222 with the largest lenses and largest supported focal lengths areimplemented using optical chains of the type shown in FIG. 5. Similarly,the five camera modules 1204, 1208, 1214, 1218, 1224 with the mediumdiameter lenses and medium supported focal lengths are also implementedusing optical chains of the type shown in FIG. 5. The five opticalchains 1226, 1228, 1230, 1232 and 1234 having the smallest diameterouter lenses and smallest focal lengths are implemented using opticalchains which do not use mirrors and extend straight toward the back ofthe camera. Optical chains of the type used in the FIG. 3 embodiment maybe used for the optical chains 1226, 1228, 1230, 1232 and 1234. However,it should be appreciated that optical chains of the type illustrated inFIG. 5 maybe and in some embodiments are, used as the optical chains1226, 1228, 1230, 1232 and 1234.

From the FIG. 12A example which may be considered as a frontal view withthe front of the camera housing removed to allow viewing of the cameramodules, it can be seen how a larger number of camera modules can beincorporated into a single camera device 600 allowing for thesimultaneous and/or synchronized capture of multiple images of the sameor different portions of a scene area using a single camera. The cameradevice can then combine multiple images to generate a composite imagehaving image attributes and/or qualities such as a number of pixelswhich exceeds that possible using a single one of the camera modules ofthe camera 600.

FIG. 12B illustrates a perspective view 1250 of the camera device 600showing the arrangement of various optical chains in the camera deviceand the elements of the optical chains in the camera device in greaterdetail. Thus FIG. 12B presents a more detailed illustration of theplurality of optical chains (OCs) 1202, 1204, 1206, 1208, 1210, 1212,1214, 1216, 1218, 1220, 1222, 1224, 1226, 1228, 1230, 1232 and 1234having various corresponding focal lengths as discussed with regard toFIG. 12A in detail.

As illustrated in FIG. 12B, the camera 600 has a depth D1 whichrepresents the thickness of the camera 600 from the front surface of thecamera (indicated by arrow 1223) to the back/rear surface of the camera(indicated by arrow 1227). While not shown in the FIG. 12B in someembodiments the camera device 600 includes the same or similar elementsas the camera device of FIGS. 1 and/or 4A.

In some embodiments the elements included in the optical chains 1202,1206, 1210, 1212, 1216, 1220, 1222, 1204, 1208, 1214, 1218, 1224 aresimilar to those discussed above with regard to FIGS. 4B and 5 while theelements included in the optical chains 1226, 1228, 1230, 1232 and 1234are similar to those discussed above with regard to FIG. 3. In theembodiment of FIG. 12B each OC uses a round outer lens.

The OC 1202 includes an outer lens 1203, a light redirection device1205, e.g., mirror, positioned behind the lens 1203, a first inner lens1207, a filter 1213, a second inner lens 1215, and a sensor 1217. Insome embodiments the OCs 1202, 1206, 1210, 1212, 1216, 1220, 1222 havethe same focal length (largest focal length compared to other OCs inFIG. 12) and use similar elements such as the mirror, filter, sensoretc. Accordingly, the elements corresponding to OCs 1206, 1210, 1212,1216, 1220, 1222 have been identified using the same reference numeralsused for identifying similar elements in the OC 1202 but with thereference numbers in these OCs followed by a prime (′), double prime(″), triple prime (′″) etc. For example, OC 1206 includes an outer lens1203′, a light redirection device 1205′, e.g., mirror, positioned behindthe lens 1203′, a first inner lens 1207′, a filter 1213′, a second innerlens 1215′, and a sensor 1217′. The OC 1210 includes an outer lens1203″, a light redirection device 1205″, a first inner lens 1207″, afilter 1213″, a second inner lens 1215″, and a sensor 1217″. The OC 1212includes an outer lens 1203′″, a light redirection device 1205″, a firstinner lens 1207″, a filter 1213′″, a second inner lens 1215″, and asensor 1217″. The OC 1216 includes an outer lens 1203″″, a lightredirection device 1205″″, a first inner lens 1207″″, a filter 1213″″, asecond inner lens 1215″″, and a sensor 1217″″. The OC 1220 includes anouter lens 1203′″″, a light redirection device 1205′″″, a first innerlens 1207′″″, a filter 1213′″″, a second inner lens 1215′″″, and asensor 1217′″″. The OC 1222 includes an outer lens 1203″″″, a lightredirection device 1205″″″, a first inner lens 1207″″″, a filter1213″″″, a second inner lens 1215″″″, and a sensor 1217″″″.

Similarly the elements corresponding to OCs 1204, 1208, 1214, 1218, 1224which have the same focal lengths (intermediate) have been identifiedusing the same reference numerals. The OC 1204 includes an outer lens1233, a light redirection device 1235, e.g., mirror, positioned behindthe lens 1233, a first inner lens 1237, a filter 1243, a second innerlens 1245, and a sensor 1247. Optical chain 1208 includes an outer lens1233′, a light redirection device 1235′, e.g., mirror, positioned behindthe lens 1233′, a first inner lens 1237′, a filter 1243′, a second innerlens 1245′, and a sensor 1247′. OC 1214 includes an outer lens 1233″, alight redirection device 1235″, a first inner lens 1237″, a filter1243″, a second inner lens 1245″, and a sensor 1247″. OC 1218 includesan outer lens 1233′″, a light redirection device 1235′″, a first innerlens 1237″, a filter 1243′″, a second inner lens 1245′″, and a sensor1247′″ and the OC 1224 includes an outer lens 1233″″, a lightredirection device 1235″″, a first inner lens 1237″″, a filter 1243″″, asecond inner lens 1245″″, and a sensor 1247″″.

As discussed with regard to FIG. 4B, an optical chain such as theoptical chain 1202 (or OCs 1206, 1210, 1212, 1216, 1220, 1222, 1204,1208, 1214, 1218, 1224), that has a light redirection element, such asthe element 1205, can be divided, for purposes of discussion, into twoparts. The optical axis of the optical chain 1202 as seen from outsideof the front of the camera is the optical axis of a first part 1201(entering the OC from the front 1223 of the camera 600 via the outerlens 1203). Light traveling into the optical chain 1202 along theoptical axis is redirected by the redirection element 1205 and traversesa second part 1209 of the first optical chain and reaches the sensor1217. Similarly, the optical axis of the optical chain 1204 includes afirst part 1211 and a second part 1219 after light redirection by theredirection element 1235, the optical axis of the optical chain 1206includes a first part 1221 and a second part 1229, the optical axis ofthe optical chain 1208 includes a first part 1231 and a second part1239, the optical axis of the optical chain 1210 includes a first part1241 and a second part 1249, the optical axis of the optical chain 1212includes a first part 1251 and a second part 1259, the optical axis ofthe optical chain 1214 includes a first part 1261 and a second part1269, the optical axis of the optical chain 1216 includes a first part1271 and a second part 1279, the optical axis of the optical chain 1218includes a first part 1278 and a second part 1288, the optical axis ofthe optical chain 1220 includes a first part 1281 and a second part1289, the optical axis of the optical chain 1222 includes a first part1291 and a second part 1299, and the optical axis of the optical chain1224 includes a first part 1292 and a second part 1298.

The other optical chains OCs 1226, 1228, 1230, 1232 and 1234 (smallestfocal length OCs) while each having an outermost lens 1252, 1253, 1254,1255, and 1256 respectively through which light enters, the OCs 1226,1228, 1230, 1232 and 1234 do not have light redirection elements in theFIG. 12B example. While not shown in FIG. 12B the OCs 1226, 1228, 1230,1232 and 1234 each has an optical axis which is perpendicular to thefront face 1223 of the camera 600.

The function of the various elements of an OC such as the outer andinner lenses, mirror, filters and sensors, has been discussed earlier,for example in the discussion of FIGS. 4B and 5. Since the function ofthe elements of the OCs shown in FIG. 12B is the same or similar to thatdiscussed with regard to FIGS. 4A-4B and 5, the discussion will not berepeated.

Light enters each of the OCs 1202, 1206, 1210, 1212, 1216, 1220, 1222,1204, 1208, 1214, 1218, 1224 via their respective outer lenses and isredirected by their respective redirection elements so that it reachesthe respective sensors at the back of each of the optical chains. Inmany cases the outer lens through which the light enters the OC isreferred to as the entrance pupil via which the light enters. Forexample, light entering through outer lens 1203 of the optical chain1202 (e.g., from the front 1223 of the camera 600 as indicated by thefirst optical axis 1201) is redirected by mirror 1205 so that it passesthrough the first inner lens 1207, the filter 1213 and the second innerlens 1215 as it travels towards sensor 1217. More or less number ofelements, e.g., lenses, filters etc., may be included in each of the OCsin some embodiments. Different optical chains may use different lenseswhile still using a sensor of the same shape and/or resolution as theother optical chains in the camera device 600.

It should be appreciated that the light redirection elements, e.g., suchas a hinged mirror or other light redirection device such as a prism,positioned behind the lens of an OC can be moved and/or rotated whichresults in changing of the optical axis of the OC seen from outside theouter lens of the corresponding OC. That is the optical axis of anoptical chain as seen from outside the camera (discussed above as theoptical axis of a first part such as optical axes 1201, 1211, 1231 etc.)can be changed by controlling the light redirection elements of thecorresponding OC. Thus it should be appreciated that while in FIG. 12Bexample the optical axes 1201, 1211, 1221, 1231, . . . 1298, 1299 appearto be parallel, in some embodiments by controlling the light redirectionelement such as the mirror placed behind the outer lens in thecorresponding optical chains, the optical axes can be changed such thatthe optical axes of one or more OCs are not parallel to each other. Theability to change the optical axis of the optical chain by controllingthe movement of a mirror, provides the same effect as if the camera isbeing pointed in a given direction, e.g., to take pictures in thatdirection, and thus provides the ability to capture images fromdifferent perspectives without actually moving the camera device.

FIGS. 12C through 12E which show optical chain 1210 with the lightredirection device 1205″ at different positions is exemplary of how thefirst part 1241 of an optical path may be altered by altering theposition of the light redirection device 1205″. The OC 1210 includes anouter lens 1203″, a light redirection device 1205″, e.g., mirror,positioned behind the lens 1203″, a first inner lens 1207″, a filter1213″, a second inner lens 1215″, and a sensor 1217″.

In FIG. 12C the angle of the mirror is inclined upward resulting in thefirst portion 1241 of the optical path to be angled to the left. Asshould be appreciated by raising the mirror 1205″ to this position adifferent image area will be observed and captured by the sensor 1217″than when the light redirection device 1205″ is inclined at 45 degreesrelative to the bottom of the optical chain module 1210, which willcoincide to the rear of the camera in which the module is mountedassuming the lens 1203″ is facing the front of the camera.

Assuming that in the FIG. 12D configuration, i.e., when the plain mirror1205″ is at a 45 degree angle and the image directed to the sensorcorresponds to the center portion of a scene area, changing theposition, e.g., angle, of the mirror 1205″ will alter the portion of ascene area captured by the sensor 1217″. FIG. 12E illustrates how thefirst portion 1241 can be directed to the right by lowering the angle ofmirror 1205″ while FIG. 12F shows how the portion of the scene whichwill be captured and be changed further by lowering the angle of themirror 1205″ even further.

While raising the angle of the light redirection device as shown in FIG.12C is supported in some embodiments, in other embodiments angles of 45degrees and less are supported. In such embodiments the maximum depth ofthe camera need not be increased to support the ability to raise thelight redirection device beyond the 45 degree angle.

It should be appreciated that by altering the position of the lightredirection devices of the modules shown in FIG. 12B the image areascaptured by the modules with light redirection devices can be changedwithout moving the module itself. For optical chain modules arrangedparallel to a 45 degree angle across the face of the camera, such asmodules 1208, 1210, 1212, 1214, 1218, 1220, 1202, 1204, changing theangle of the light redirection device effectively moves the imagecapture area, e.g., from directly in front of the center of the cameraor from capturing a corner quadrant of a scene area of interest forexample, to some other portion of the scene area. In cases where thecamera module is arranged along or parallel the 45 degree angleextending form one corner of the front face of the camera device 1250 toanother corner of the camera device 1250 changing the mirror positionwill have the effect of sliding the image area being captured along the45 degree diagonal. Thus, when the camera modules are located along the45 degree bias of the front of the camera, the image capture area may beshifted so that a quadrant of a large image area is captured rather thana center portion of the large image area via simple movement of thelight redirection devices of modules arranged along the 45 degree anglewith respect to the front face of the camera. The images capture areamay be effectively slide from the center to a corner quadrant, or viseversa, by altering the angle of the light rejection device 1205″ asshown in FIGS. 12C through 12D.

Different optical chains, depending on their mirror positions and focallength, can, and in some embodiments are, used to capture differentportions of a scene area of interest. The image areas which are capturedmay, and often do, overlap but need not overlap. Overlap of areascaptured by different camera modules facilities combining of the imagescaptured by the different modules.

For example consider the following discussion regarding one embodimentwhere the optical axes of at least some optical chains are not parallel.While for discussion purposes we refer to FIG. 12B and use the referencenumbers corresponding to the elements shown in FIG. 12B, it isunderstood for the discussion of this particular embodiment that unlikethe FIG. 12B example, the optical axes of two or more optical chains (asseen from outside the outermost lens of the optical chains) are notparallel to each other and may not be perpendicular to the front face ofthe camera. A camera device implemented in accordance with thisembodiment includes a first optical chain, e.g., a first OC 1202, of acamera having a first optical axis and a first outermost lens and asecond optical chain, e.g., a second OC 1208 or OC 1210, of the camerahaving a second optical axis which is not parallel to the first opticalaxis and a second outermost lens which is different from the firstoutermost lens. The first optical chain, can be and in some embodimentsis, used to capture a first image of a first portion of a scene area ofinterest and the second optical chain, can be and in some embodimentsis, used to capture a second image of a second portion of the scene areaof interest. In some such embodiments the first and second optical axesare not perpendicular to the front face of the camera, e.g., such asfront (1223) face of camera 600. In some embodiments in addition to thefirst and second optical chains, the exemplary camera includes a thirdoptical chain, e.g., OC 1228, having a third optical axis and a thirdoutermost lens 1253 which is separate from the first and secondoutermost lenses (e.g., 1203 and 1203″). In some embodiments the thirdoptical chain has a focal length which is smaller than a focal length ofat least one of the first or second optical chains. In some embodimentsthe third optical chain is used to capture a third image, the thirdimage including the scene area of interest, e.g., image of the entirescene area of interest. In some embodiments the third optical axis isperpendicular to the front face 1223 of the camera. In some embodimentsthe third optical axis is not parallel to either of the first optical orthe second optical axis. In some embodiments the second optical chainhas a second focal length and the first optical chain has a first focallength, the second focal length is smaller than said first focal length.In some other embodiments the first and second focal lengths are thesame. In some embodiments the third optical chain has a third focallength which is smaller than the first and second focal lengths.

In some embodiments in addition to the first, second and third opticalchains, the camera includes a fourth optical chain, e.g., OC 1234,having a fourth outermost lens, e.g., lens 1256, which is separate fromsaid first, second and third outermost lenses. In some embodiments thefourth optical chain is used to capture a fourth image, the fourth imageincluding a second image of the scene area of interest, e.g., entirescene area of interest. In some embodiments the fourth optical chain hasa fourth optical axis, and the third and fourth optical axes areparallel to each other. In some embodiments the third and fourth opticalaxes are not parallel to the first or second optical axis.

In some embodiments the camera further includes a fifth optical chain,e.g., OC 1220, having a fifth optical axis which is not parallel to thefirst and second optical axes and a sixth optical chain, e.g., OC 1212,having a sixth optical axis which is not parallel to the first, second,or fifth optical axis. In some embodiments the fifth optical chain isused to capture a fifth image and the sixth optical chain is used tocapture a sixth image, the fifth image being an image of a third portionof the scene area of interest and the sixth image being an image of afourth portion of the scene area of interest. In some embodiments thecamera further includes a seventh optical chain, e.g., OC 1216, havingthe same focal length as the first optical chain, a seventh image, saidseventh optical chain having an optical axis perpendicular to the face(1223) of the camera.

In some embodiments the camera 600 includes a processor (e.g., processor110, 211) configured to generate a composite image by combining at leastthe first and second images. In some embodiments the processor isconfigured to generate the composite image from the first, second,third, fourth, fifth and sixth images. In some embodiments the processoris configured to generate the composite image from the first, second,third, fourth, fifth, sixth and seventh images. In some embodiments theprocessor is further configured to control storage of the generatedcomposite image in the device memory, e.g., memory 108, and/or output ofthe composite image on a display, e.g., display 102, and/or transmissionof the captured images or the composite image to another device via aninterface such as interface 114.

FIG. 13A shows an exemplary optical chain, e.g., a camera module, 1300which is similar in construction to the camera module 500 shown in FIG.5 but which uses a non-round, e.g., oval, outer lens 1312 and arectangular mirror 1310 as a light redirection device. The optical chain1300 is used in various exemplary embodiments. The camera module 1300 isan optical chain which includes an outer lenses 1312 which, in some butnot all embodiments, is plane glass with no optical power, a lightredirection device, e.g., mirror, 1310 positioned behind the lens 1312,a hinge drive 1316, a mirror hinge 1308, a first cylindrical moduleportion 1306, a second cylindrical module portion 1304, a sensor 1302and a lens drive 1313. In some embodiments the cylindrical moduleportions may not be exactly cylindrical as the cross section in someembodiments maybe oval rather than circular. However even with ovalcross sections the portions appear generally cylindrical in shape andthus will be referred to as cylindrical portions. Light enters thecamera module 1300 via the lens 1312 and is redirected by the mirror1310 so that it reaches the sensor 1302 at the back of the opticalchain. The first and second cylindrical module portions 1304, 1306 canhouse one or more lenses or filters as well as other optical componentsthrough which light may pass before reaching the sensor 1302. While theouter lens 1312 has a non-round aperture, the sensor 1302 may be thesame shape and/or type as used in the FIG. 5 embodiment. Thus whiledifferent optical chains may use different lenses they may still use asensor of the same shape and/or resolution as the other optical chainsin the camera device.

The advantages of using an optical chain having a non-round aperture interms of potential camera thickness will be apparent in view of thediscussion which follows.

FIG. 13B illustrates an exemplary arrangement of a plurality of opticalchains (OCs) in a camera device 1320 including a camera housing 1322. Inaddition to the arrangement of the optical chains inside the camera, theconfiguration and arrangement of the internal elements of each of theoptical chains is also shown in greater detail. While one exemplaryarrangement with six OCs is shown in FIG. 13B, it should be appreciatedthat the optical chains can be arranged in various other positionswithin the camera 1320. As illustrated in FIG. 13B, the camera 1320 hasa depth D which represents the thickness of the camera 1320 from thefront surface of the camera (indicated by arrow 1321) to the back/rearsurface of the camera (indicated by arrow 1325). It should beappreciated that FIG. 13B shows a perspective view of camera device 1320so that various features of the camera device 1320 such as camerahousing 1322, thickness (D) 1323, front and rear surfaces 1321 and 1325can be better appreciated. In some embodiments the camera device 1320includes the same or similar elements as the camera device of FIGS. 1and/or 4A.

The plurality of optical chains shown in FIG. 13B includes OC 1324, OC1326, OC 1328, OC 1330, OC 1332 and OC 1333. The elements included ineach of the various optical chains shown are similar to those discussedabove with regard to FIG. 4A and FIG. 13A. In the embodiment of FIG. 13Beach OC uses a non-round, e.g., oval, outer lens.

The OC 1324 includes an outer lens 1334, a light redirection device1335, e.g., mirror, positioned behind the lens 1334, a first inner lens1336, a filter 1338, a second inner lens 1340, and a sensor 1342. The OC1326 includes an outer lens 1344, a light redirection device 1345positioned behind the lens 1344, a first inner lens 1346, a filter 1348,a second inner lens 1350, and a sensor 1352. The OC 1328 includes anouter lens 1354, a light redirection device 1355 positioned behind thelens 1354, a first inner lens 1356, a filter 1358, a second inner lens1360, and a sensor 1362. Similarly, OC 1330 includes an outer lens 1364,a light redirection device 1365 positioned behind the lens 1364, a firstinner lens 1366, a filter 1368, a second inner lens 1370, and a sensor1372. The OC 1332 includes an outer lens 1374, a light redirectiondevice 1375 positioned behind the lens 1374, a first inner lens 1376, afilter 1378, a second inner lens 1380, and a sensor 1382. The OC 1333includes an outer lens 1384, a light redirection device 1385 positionedbehind the lens 1384, a first inner lens 1386, a filter 1388, a secondinner lens 1390, and a sensor 1392.

As discussed with regard to FIG. 4B, an optical chain such as theoptical chain 1324 shown in FIG. 13B, that has a light redirectionelement, such as the element 1335, can be divided, for purposes ofdiscussion, into two parts. The optical axis of the optical chain 1324as seen from outside the camera is the optical axis of a first part 1331(entering the OC from the front of the camera 1320 via the outer lens1334). Light traveling into the optical chain 1324 along the opticalaxis is redirected by the redirection element 1335 and traverses asecond part 1339 of the first optical chain and reaches the sensor 1342.Similarly, the optical axis of the optical chain 1326 includes a firstpart 1341 and a second part 1349 after light redirection by theredirection element 1345, the optical axis of the optical chain 1328includes a first part 1351 and a second part 1359, the optical axis ofthe optical chain 1330 includes a first part 1361 and a second part1369, the optical axis of the optical chain 1332 includes a first part1371 and a second part 1379 and the optical axis of the optical chain1333 includes a first part 1381 and a second part 1389.

In some particular embodiments, the camera 1320 has a plurality ofoptical chains/camera modules such as the OCs 1324 through 1333 havinglenses with non-circular (also referred as non-round) apertures. In onesuch embodiment the camera 1320 includes a first optical chain, e.g., OC1324, that uses the lens 1336 whose non-circular aperture has a lengthless than or equal to D (camera thickness) in a first direction alongthe direction of the thickness of the camera and a length larger than Dalong a second direction perpendicular to the direction of the thicknessof the camera. In various embodiments the first lens is the lens closestto the light redirection element on the second optical axis portion,e.g., lens 1336. In some such embodiments the camera 1320 furtherincludes a second optical chain, e.g., OC 1326, that uses a lens 1346whose non-circular aperture has a length less than or equal to D in thefirst direction along the direction of the thickness of the camera and alength larger than D along a third direction perpendicular to the firstdirection. In some such embodiments the second and third directions areat an angle of 90 degrees with respect to each other. In some suchembodiments the camera 1320 further includes a third optical chain,e.g., OC 1333, that uses the lens 1386 whose non-circular aperture has alength less than or equal to D in a first direction along the directionof the thickness of the camera and a length larger than D along a fourthdirection perpendicular to the first direction, the first, second andthird directions being different. In some such embodiments the secondand third directions are at an angle of 90 degrees with respect to eachother and the second and fourth direction are at an angle between 30degrees and 60 degrees (e.g., 45 degrees) with respect to each other. Insome embodiments the first light redirection device is a mirror. In someembodiments the first light redirection device is a prism. In someembodiments the first light redirection device is a 45 degree angledplane mirror which redirects light by 90 degrees.

The function of the various elements of an OC such as the outer andinner lenses, mirror, filters and sensors, has been discussed earlier,for example in the discussion of FIGS. 4A and 13A. Since the function ofthe elements of the OCs shown in FIG. 13B is the same or similar andthus the discussion will not be repeated in detail.

Light enters each of the OCs 1324 through 1333 via their respectiveouter lenses and is redirected by their respective redirection elementsso that it reaches the respective sensors at the back of each of theoptical chains. In many cases the outer lens through which the lightenters the OC is referred to as the entrance pupil via which the lightenters. For example, light entering through outer lens 1334 of theoptical chain 1324 (e.g., from the front of the camera 1320 as indicatedby the first optical axis 1331) is redirected by mirror 1335 so that itpasses through the first inner lens 1336, the filter 1338 and the secondinner lens 1340 as it travels towards sensor 1342. More or less numberof elements, e.g., lenses, filters etc., may be included in each of theOCs in some embodiments. Different optical chains may use differentlenses while still using a sensor of the same shape and/or resolution asthe other optical chains in the camera device.

FIG. 14 shows a camera device 1400 which is similar to the camera device600 but which uses lenses with non-round apertures for large focallength optical chains. In various embodiments the lenses with non-roundapertures are included as part of various optical chains in a camerahousing 1402. The use of lenses with non-round, e.g., oval, apertures isvisible in lens area 1404. While the non-round lenses have the samemaximum length in one dimension as the round lenses used in FIG. 6 theyare smaller in the other direction, e.g., the direction perpendicular tothe direction of maximum aperture extent. Different orientations of thelenses with the non-round apertures is intentionally used to allow forthe capture of high frequency information in addition to low frequencyinformation in a variety of directions. The camera 1400 is similar tothe camera 1320 illustrated in FIG. 13B, however in FIG. 13B aparticular arrangement of the optical chains/camera modules withnon-round apertures is illustrated while in FIG. 14 the camera 1400includes optical chains some of which use lenses with round aperturesand some other use lenses with non-round apertures as can be seen. Theplurality of various optical chains illustrated in FIG. 13B anddiscussed above can be used in the camera 1400.

While not shown in FIG. 14, it should be appreciated that camera device1400 may and sometimes does include the same or similar elements ascamera device 100 and camera device 200 of FIGS. 1 and 4A. Thus itshould be appreciated that camera device 1400 includes various elementssuch as the processor 110/211, memory 108/213, zoom controller 140/214,exposure and read out controller 150, accelerometer, gyro, autofocuscontroller 152 etc., and various other elements discussed above withregard to camera devices 100 and 200.

FIG. 15 is a simplified frontal view 1500 of the camera 1400 with thelens area 1404 clearly shown. As can be seen more clearly in FIG. 15,the lens area 1404 of the camera shows outer lens apertures some ofwhich are round while various others are non-round. As discussed above,such a combination allows for capturing of high frequency information inaddition to low frequency information in a various directions.

FIG. 16 is a side view 1600 of the camera device 1400. In the side view1600 of the camera 1400 the front surface 1602 and rear surface 1604 ofthe camera 1400 is also shown. Note that while the camera height H isthe same as in the FIG. 10 example which corresponds to the cameradevice 600, the camera device 1400 is implemented with a depth D2 whichis less than D1 and which, in some embodiments, is less than the lengthof the oval apertures in the direction of maximum extent. In the FIG. 14embodiment by using lenses with oval apertures for the camera moduleshaving the largest focal length, the camera depth is not constrained bythe maximum dimension of the largest aperture dimension and can be lessthan the maximum length the of the aperture.

While large or the largest aperture lenses included in a camera such asthe camera 1400 may affect the minimum thickness of a camera due to thephysical size of the large lenses or light redirection devicesassociated therewith, it is often possible to support smaller aperturesusing round lenses while using the non-round lenses for the largerapertures. This is because the light redirection devices correspondingto the smaller round lenses may not be as large and deep as mirrorswhich would otherwise be required to redirect light of larger roundapertures within the body of the camera by 90 degrees. In someembodiments a combination of large lenses with non-round apertures andsmaller lenses with round apertures is used. This approach is used insome embodiments including the one shown in FIG. 17.

FIG. 17A shows an arrangement 1700 of optical chains, e.g., cameramodules which may be used to implement the camera device 1400. Theoptical chains (OCs) of the type shown in FIGS. 5 and 13A may be used inthe FIG. 17 embodiment but the implementation is not limited to the useof such optical chains. Note that lenses with round apertures are usedfor the optical chains 1704, 1708, 1714, 1718, 1724 having medium focallengths and optical chains 1726, 1728, 1730, 1732 and 1734 having smallfocal lengths. Lenses having non-round apertures are used for theoptical chains 1702, 1706, 1710, 1712, 1716, 1720, 1722, having largefocal lengths. The particular arrangement and angle of the non-roundapertures can be beneficial as will be apparent from the discussion ofFIGS. 18 to 23. Each of the optical chains 1704, 1708, 1714, 1718, and1724 use lenses with round apertures and include elements which are thesame or similar to the elements discussed in detail with regard to FIG.5. Similarly, each of the optical chains 1702, 1706, 1710, 1712, 1716,1720, 1722 use lenses with non-round apertures and include elementswhich are the same or similar to the elements discussed in detail withregard to FIG. 13A.

In one embodiment, an optical chain (OC) such as OC 1722, is a firstoptical chain having a first focal length and a first non-circular lens,and another optical chain, e.g., OC 1706, is a second optical chainhaving a second focal length and a second non-circular lens. In someembodiments the first and second focal lengths are the same. In someother embodiments the first and second focal lengths are different. Insome embodiments the first non-circular lens extends in the firstdirection by an amount which is greater than a depth of the camera 1400,e.g., depth D2 shown in FIG. 16. In some embodiments the first opticalchain includes a first sensor, and the second optical chain includes asecond sensor, e.g., such as sensor 1302 of FIG. 13A. In someembodiments the camera 1400 further includes a processor, e.g., such asprocessor 110 or 211, coupled to the first and second sensors forcombining images captured by the first and second sensors to generate acomposite image. In some embodiments the first optical chain includes afirst light redirection element and the second optical chain includes asecond light redirection element, e.g., such as the light redirectionelement 1310 of FIG. 13A. In some such embodiments the first lightredirection element extends in at least one direction by an amount whichis greater than the depth of the camera, and the second lightredirection element extends in at least one direction by an amount whichis greater than the depth of the camera. In some embodiments the firstand second light redirection elements are plane mirrors. In someembodiments the first and second light redirection elements are prisms.In some embodiments the first and second focal lengths are equal to orgreater than 70 mm and the third focal length is less than 70 mm. Insome embodiments the first and second sensors have the same number ofpixels, and the third optical chain includes a third sensor having thesame number of pixels as the first and second sensors.

In some embodiments an optical chain using round aperture lens, e.g., OC1704 or 1728, is a third optical chain having a third focal length andincluding a round lens, the third focal length being less than the firstor second focal lengths. In some embodiments the first non-circularlens, corresponding to the first optical chain 1722, is longer in afirst direction than in a second direction which is perpendicular to thefirst direction, and the second non-circular lens, corresponding to thesecond optical chain 1706, is longer in a third direction than in afourth direction, the fourth direction being perpendicular to the thirddirection. In some embodiments the first and third directions aredifferent. In some embodiments the first direction extends in a firstplane and the third direction is in the same plane as the first plane.In some embodiments the plane corresponds to a front of the camera 1400.

In some embodiments another camera module, e.g., 1710, is a fourthoptical chain having a fourth focal length and a third non-circularlens, the fourth focal length being larger than the third focal length.In some such embodiments the third non-circular lens is longer in afifth direction than in a sixth direction, the sixth direction beingperpendicular to the fifth direction. In some such embodiments thefirst, third, and fifth directions are different by at least 20 degrees.

FIG. 17B illustrates a perspective view 1750 of the camera device 1400,with the arrangement of various optical chains and elements of theoptical chains in the camera device shown in greater detail.

FIG. 17B illustrates a perspective view 1750 of the camera device 1400showing the arrangement of various optical chains in the camera deviceand the elements of the optical chains in the camera device in greaterdetail. Thus FIG. 17B presents a more detailed illustration of theplurality of optical chains (OCs) 1702, 1704, 1706, 1708, 1710, 1712,1714, 1716, 1718, 1720, 1722, 1724, 1726, 1728, 1730, 1732 and 1734having various corresponding focal lengths as discussed with regard toFIG. 17A in detail.

As illustrated in FIG. 17B, the camera 1400 has a depth D2 whichrepresents the thickness of the camera 1400 from the front surface(indicated by arrow 1723) of the camera to the back/rear surface of thecamera (indicated by arrow 1727). While not shown in the FIG. 17B insome embodiments the camera device 1400 includes the same or similarelements as the camera device of FIGS. 1 and/or 4A.

In some embodiments the elements included in the optical chains 1702,1706, 1710, 1712, 1716, 1720, 1722, are similar to those discussed abovewith regard to FIG. 13B. The optical chains 1704, 1708, 1714, 1718, 1724and the elements included in these optical chains are similar to thosediscussed with regard to OCs 1204, 1208, 1214, 1218, 1224 discussedabove with regard to FIG. 12B while the optical chains 1726, 1728, 1730,1732 and 1734 and the elements included therein are similar to thosediscussed with regard to OCs 1226, 1228, 1230, 1232 and 1234 and the OCsdiscussed with regard to FIG. 3. In the illustrated embodiment of FIG.17B some of the OCs use a non-circular outer lens while other OCs useround outer lenses.

The OC 1702 includes a non circular outer lens 1703, a light redirectiondevice 1705, e.g., mirror, positioned behind the lens 1703, a firstinner lens 1707 (non circular), a filter 1713, a second inner lens 1715,and a sensor 1717. In some embodiments the OCs 1702, 1706, 1710, 1717,1716, 1720, 1722 have the same focal length (largest focal lengthcompared to other OCs in FIG. 17) and use similar elements such as themirror, filter, sensor etc. Accordingly, the elements corresponding toOCs 1706, 1710, 1717, 1716, 1720, 1722 have been identified using thesame reference numerals used for identifying similar elements in the OC1702 but with the reference numbers in these OCs followed by a prime(′), double prime (″), triple prime (m) etc. For example, OC 1706includes a non circular outer lens 1703′, a light redirection device1705′, e.g., mirror, positioned behind the lens 1703′, a first noncircular inner lens 1707′, a filter 1713′, a second inner lens 1715′,and a sensor 1717′. The OC 1710 includes a non circular outer lens1703″, a light redirection device 1705″, a first non circular inner lens1707″, a filter 1713″, a second inner lens 1715″, and a sensor 1717″.The OC 1712 includes a non circular outer lens 1703′″, a lightredirection device 1705″, a first non circular inner lens 1707′″, afilter 1713′″, a second inner lens 1715′″, and a sensor 1717′″. The OC1716 includes a non circular outer lens 1703″″, a light redirectiondevice 1705″″, a first non circular inner lens 1707″″, a filter 1713″″,a second inner lens 1715″″, and a sensor 1717″″. The OC 1720 includes anon circular outer lens 1703′″″, a light redirection device 1705′″″, afirst non circular inner lens 1707′″″, a filter 1713′″″, a second innerlens 1715′″″, and a sensor 1717′″″. The OC 1722 includes a non circularouter lens 1703″″″, a light redirection device 1705″″″, a first noncircular inner lens 1707″″″, a filter 1713″″″, a second inner lens1715″″″, and a sensor 1717″″″.

In some embodiments the optical chains 1704, 1708, 1714, 1718, 1724 havethe same focal lengths (intermediate). The OC 1704 includes an outerlens 1733, a light redirection device 1735, e.g., mirror, positionedbehind the lens 1733, a first inner lens 1737, a filter 1743, a secondinner lens 1745, and a sensor 1747. The elements corresponding to OCs1708, 1714, 1718, 1724 which have the same focal length as OC 1704 havebeen identified using the same reference numerals used for identifyingsimilar elements in the OC 1704 but with the reference numbers in theseOCs followed by a prime (′), double prime (″), etc. As shown, opticalchain 1708 includes an outer lens 1733′, a light redirection device1735′, e.g., mirror, positioned behind the lens 1733′, a first innerlens 1737′, a filter 1743′, a second inner lens 1745′, and a sensor1747′. OC 1714 includes an outer lens 1733″, a light redirection device1735″, a first inner lens 1737″, a filter 1743″, a second inner lens1745″, and a sensor 1747″. OC 1718 includes an outer lens 1733′″, alight redirection device 1735′″, a first inner lens 1737′″, a filter1743′″, a second inner lens 1745′″, and a sensor 1747′″ and the OC 1724includes an outer lens 1733″″, a light redirection device 1735″″, afirst inner lens 1737″″, a filter 1743″″, a second inner lens 1745″″,and a sensor 1747″″.

As discussed earlier, an optical chain such as the OC 1702 (or OCs 1706,1710, 1712, 1716, 1720, 1722, 1704, 1708, 1714, 1718, 1724), that has alight redirection element, such as the element 1705, can be divided, forpurposes of discussion, into two parts. The optical axis of the opticalchain 1702 as seen from outside of the front of the camera is theoptical axis of a first part 1701 (entering the OC from the front 1723of the camera 1400 via the outer lens 1703). Light traveling into theoptical chain 1702 along the optical axis is redirected by theredirection element 1705 and traverses a second part 1709 of the firstoptical chain and reaches the sensor 1717. Similarly, the optical axisof the optical chain 1704 includes a first part 1711 and a second part1719 after light redirection by the redirection element 1735, theoptical axis of the optical chain 1706 includes a first part 1721 and asecond part 1729, the optical axis of the optical chain 1708 includes afirst part 1731 and a second part 1739, the optical axis of the opticalchain 1710 includes a first part 1741 and a second part 1749, theoptical axis of the optical chain 1717 includes a first part 1751 and asecond part 1759, the optical axis of the optical chain 1714 includes afirst part 1761 and a second part 1769, the optical axis of the opticalchain 1716 includes a first part 1771 and a second part 1779, theoptical axis of the optical chain 1718 includes a first part 1778 and asecond part 1788, the optical axis of the optical chain 1720 includes afirst part 1781 and a second part 1789, the optical axis of the opticalchain 1722 includes a first part 1791 and a second part 1799, and theoptical axis of the optical chain 1724 includes a first part 1792 and asecond part 1798.

The other optical chains OCs 1726, 1728, 1730, 1732 and 1734 (smallestfocal length OCs) while each having an outermost lens 1752, 1753, 1754,1755, and 1756 respectively through which light enters, the OCs 1726,1728, 1730, 1732 and 1734 do not have light redirection elements in theFIG. 17B example. While not shown in FIG. 17B, the OCs 1726, 1728, 1730,1732 and 1734 each has an optical axis which is perpendicular to thefront face 1723 of the camera 1400.

The function of the various elements of an OC such as the outer andinner lenses, mirror, filters and sensors, has been discussed earlier,for example with regard to FIGS. 4B and 5. Since the function of theelements of the OCs shown in FIG. 17B is the same or similar to thatdiscussed with regard to FIGS. 4A-4B and 5, the discussion will not berepeated.

Light enters each of the OCs 1702, 1706, 1710, 1717, 1716, 1720, 1722,1704, 1708, 1714, 1718, 1724 via their respective outer lenses and isredirected by their respective redirection elements so that it reachesthe respective sensors at the back of each of the optical chains. Inmany cases the outer lens through which the light enters the OC isreferred to as the entrance pupil via which the light enters. Forexample, light entering through outer lens 1703 of the optical chain1702 (e.g., from the front 1723 of the camera 1400 as indicated by thefirst optical axis 1701) is redirected by mirror 1705 so that it passesthrough the first inner lens 1707, the filter 1713 and the second innerlens 1715 as it travels towards sensor 1717. More or less number ofelements, e.g., lenses, filters etc., may be included in each of the OCsin some embodiments. Different optical chains may use different lenseswhile still using a sensor of the same shape and/or resolution as theother optical chains in the camera device 1400. It should be appreciatedthat the use of optical chains with non-circular lenses arranged in themanner illustrated in FIG. 17B provides several advantages and allowsimplementation of a thin camera with optical chains have variousdifferent focal lengths.

It should be appreciated that the light redirection elements, e.g., suchas a hinged mirror, positioned behind the lens of an OC can be movedand/or rotated which results in changing of the optical axis of the OCseen from outside the outer lens of the corresponding OC. That is theoptical axis of an optical chain as seen from outside the camera(discussed above as the optical axis of a first part such as opticalaxes 1701, 1711, 1731 etc.) can be changed by controlling the lightredirection elements of the corresponding OC. Thus it should beappreciated that while in FIG. 17B example the optical axes 1701, 1711,1721, 1731, . . . 1798, 1799 appear to be parallel, in some embodimentsby controlling the light redirection element such as the mirror placedbehind the outer lens in the corresponding optical chains, the opticalaxes can be changed such that the optical axes of one or more OCs arenot parallel to each other. The ability to change the optical axis ofthe optical chain by controlling the movement of a mirror, provides thesame effect as if the camera is being pointed in a given direction,e.g., to take pictures in that direction, and thus provides the abilityto capture images from different perspectives without actually movingthe camera device.

FIG. 18 shows a round aperture 1800 corresponding to an exemplary lenswith a round opening such as the lenses which may and sometimes are usedin the FIGS. 12A and 12B embodiment.

FIG. 19 shows the possible frequency characteristics which are expectedfrom a well designed (diffraction limited) lens with a round aperture1800 of the type shown in FIG. 18 with the frequency information beinguniform in both vertical and horizontal directions (actually anydirection). The uniform frequency characteristics in the vertical andhorizontal dimension will allow for the same or similar frequencycharacteristics and thus resolutions in both the vertical and horizontaldimensions. While the number of pixels or pixel elements of a sensor canaffect resolution, the number of picture elements of a sensor or imageshould not be confused with resolution which relates to the ability of aviewer to distinguish between different features of an image. Resolutionquantifies how close lines can be to each other and still be visiblyresolved. Resolution units can be tied to physical sizes such as linesper mm, lines per inch and/or to the overall size of an image such,e.g., lines per picture height.

Given the generally uniform frequency response of a lens with a roundaperture, a view of an image captured using a round lens should be ableto identify vertical and horizontal lines equally well.

FIG. 20 shows how, in the case of a round aperture 1800, the length ofthe opening through which light passes is the same in both dimensions ofthe plane in which the lens opening exists. As shown, the length of theopening in the vertical direction L₁ equals the length of the opening inthe horizontal direction L₂. As should be appreciated this is not thecase where a lens has a non-round aperture, e.g., an oval, oblong orother non-round shaped aperture through which light passes.

FIG. 21 shows an exemplary non-round, e.g., oval, aperture 2100, withthe shading being used to show the relative amount of frequencyinformation in each of the horizontal and vertical directions which willbe captured. being clear from the figure that more frequency informationis available in the vertical direction than in the horizontal directionthereby resulting in higher frequency information being captured andavailable in the longer dimension, e.g., dimension of maximum extent, ofthe aperture than in the narrower dimension.

In general, the performance of even the best designed lenses is limitedby their aperture size because of the diffraction limit. The larger theaperture, the better a well-designed lens should be able to perform interms of capturing detail. In other words, the larger the aperture thegreater is the resolution that is possible. The captured resolution canbe expressed in the frequency domain as in FIG. 19. A larger radius ofthe circle in FIG. 19 indicates more captured high frequency informationin all spatial directions of the image, e.g., horizontal, vertical orslanted. It is important to note that even when a camera module is notdiffraction limited and the above may not apply, it is sometimes stilldesirable to use a larger aperture because a larger aperture areacaptures more light in a given period of time making the images lessgrainy or noisy when the exposure time is limited as is often the casewhen a subject which is to have its image captured is in motion.

If the aperture is not round but oval, for a well-designed diffractionlimited lens there is correspondingly more detail captured along thedirection of the longer oval dimension than the direction of the shorteroval dimension.

FIG. 22 shows a comparison of the lengths of the non-round aperture 2100in the vertical (Y) and horizontal (X) directions, with the verticaldimension being the longer of the two dimensions in the FIG. 22 example.As shown, the length of the opening through which light passes in thevertical direction L_(Y) is greater than the length of the opening inthe horizontal direction L_(X).

FIG. 23 is a diagram 2300 showing how, by combining image informationfrom multiple non-round lenses with oval apertures 2100, 2100′, 2100″,2100′″ oriented in different directions, image information approximatingthe information expected to be obtained from a lens (with radius equalto the bigger oval dimension) with a round aperture 1800 can be achievedwith more information being available towards the center of the combinedimage than at various edge locations due to the overlapping of multipleindividual images which are combined in the FIG. 23 example to generatea composite image. The capture of high frequency information through theuse of multiple non-round lenses at different angles to each other andcombining the images helps make up for the lack of high frequencyinformation captured by an individual non-round lens. It should beappreciated that while the resolution that can be achieved by combiningimages captured with lenses having apertures as shown in FIG. 23 canapproximate that of an image captured using a round lens, the compositeimage may have a larger dynamic range in terms of luminance informationin the regions of image overlap than would be obtained using a singleround lens to capture the same image. This is because the differentlenses each capture light in the center region. It should be appreciatedthat a composite image generated by combining images captured using lenshaving apertures as shown in FIG. 23, in accordance with one embodimentof the invention, will result in a high quality center image portion andlower quality towards the edges of the composite image. This imagequality fall off is similar to what might be expected if a single highquality lens were used with the image quality normally being bettertowards the center of the lens than towards the edges of the lens.

Accordingly, it should be appreciated that a composite image generatedby combining images captured using lenses with apertures as shown inFIG. 23 can result in a composite image of similar or better quality inat least some respects than might be obtained by capturing an imageusing a single round lens with a similar maximum aperture dimension in adirection of maximum extent of the aperture.

FIG. 24 shows how a lens with a round aperture 1800 can be cut or maskedto produce a lens having a non-round aperture, e.g., approximating thatof an oval or oblong shape. Such an embodiment allows for the capture ofthe most or a significant amount of light for a given max aperture.

FIG. 25 shows the aperture 2500 resulting from cutting or masking around lens, e.g., a lens with a round aperture 1800, to produce a lenshaving an aperture as shown in FIG. 24, e.g., an oblong apertureapproximating an oval aperture.

Lenses of the type shown in FIG. 25 are used as the non-round lenses ofcamera devices in some embodiments. Thus, while shown as oval lenses insome embodiments of the camera device of FIG. 14, lenses with non-ovalapertures of the type shown in FIG. 25 are used.

FIG. 26 is a diagram of a camera 2600 that includes a housing 2610having a front side 2614, top 2612, bottom 2616 and rear or back 2618. Around lens corresponding to round aperture 2606 is mounted in the front2614 of the housing 2610. FIG. 26 shows how using a light redirectiondevice 2604, e.g., mirror, to redirect light 90 degrees in combinationwith an outer lens having a round aperture 2606 normally requires acamera depth, e.g., thickness, equal to or greater than the diameter ofthe lens and corresponding aperture 2606 which is part of the lens. Insome embodiments of the invention the lens is simply a round flat pieceof plastic or glass. The round aperture 2606 is shown in FIG. 26 on thefront of the camera device 2600 through the use of dashed lines. Lightpassing through the round aperture 2606 will be reflected at 90 degreesby the mirror 2604 up towards the sensor 2602 mounted at the top of thecamera 2600. Note the front to rear distance R_(D) of the lightredirection device mounted in the camera and the height R_(H) of thelight redirection device in the FIG. 26 example are equal with bothbeing greater than or equal to the diameter L_(X) of the round aperture2606. As a result, the depth of the camera 2600 is equal to or greaterthan the diameter of the round aperture 2606.

FIG. 27 is a diagram of a camera 2700 that includes a housing 2710having a front side 2714, top 2712, bottom 2716 and rear or back 2718. Anon-round lens corresponding to non-round, e.g., oval, aperture 2706 ismounted in the front 2714 of the housing 2710. FIG. 27 shows how using alight redirection device 2704, e.g., mirror, to redirect light 90degrees in combination with an outer lens having a non-round, e.g.,oval, aperture 2706 can allow for use of lenses which are longer in onedimension than the camera is deep. Note that the sensor 2702 is mountedon the top 2712 of the camera 2700 (parallel to the top portion of thecamera housing) in the same position as the sensor 2602 in the FIG. 26embodiment but that the mirror 2704 can now be implemented using amirror with a flat rectangular surface which is less deep than themirror 2604 used in the FIG. 26 embodiment. It is also worth pointingout that while a rectangular mirror is used for simplicity, an ovalmirror can also be used and can also act as the aperture stop in someembodiments. It should be appreciated that use of non-round, e.g., oval,aperture in some embodiments allows for thinner camera, e.g., camerawith less thickness than a camera with round aperture with diameterequal to the maximum dimension of the oval aperture.

FIG. 28 is a diagram of a camera 2800 that includes a housing 2810having a front side 2814, top 2812, bottom 2816 and rear or back 2818. Anon-round lens corresponding to non-round, e.g., oval, aperture 2806 ismounted in the front 2814 of the housing 2810. FIG. 28 shows how in thecamera 2800 the length of the light path of an optical chain including anon-round aperture 2806 can be longer than the depth of the camera 2800with the light redirection device 2804 being capable of being positionedat one end, e.g., the bottom end 2816, of the camera device 2800. Itshould be appreciated that the positioning of the light redirectiondevice 2804 and the sensor 2802 near the opposite ends of the camera2800 allows for ample space for additional oval aperture lenses andlight redirection devices.

FIG. 29 shows an example of a camera 2900 in which multiple lightredirection devices 2904, 2901 are used in an optical chain, e.g.,camera module, to allow for a relatively long light travel path and thusfocal length while allowing a sensor 2902 to be positioned on either theback 2918 or front 2914 of the camera depending on which way the lightis redirected as it passes through a camera module which includes a lensmounted in the front of the camera corresponding to aperture 2906, firstlight redirection device 2904, second light redirection device 2901 andsensor 2902. Note that in the FIG. 29 outer lens of the camera module ispositioned near the bottom 2916 of the camera 2900 as is the first lightredirection device 2904 while the second light redirection device 2901is positioned near the top 2912 of the camera 2900. While the sensor2902 is shown mounted on the inside back 2918 of the camera 2900 inother embodiments, e.g., where the second light redirection device 2901directs light towards the front of the camera, the sensor 2902 ismounted on the inside front of the camera housing 2910.

FIG. 30 is an illustration of a camera device 3000 that includes 4optical chains with non-circular outer lenses and apertures. Forpurposes of simplicity as with FIGS. 27-29 in the FIG. 30 embodiment asingle reference number is used to represent both the non-circular lensand its corresponding non-circular aperture. The light enters the outernon-circular apertures of the lenses via the front of the camera and isredirected to corresponding sensors via one or more light redirectiondevices position under the outer lenses (e.g., lenses 3001, 3011, 3021and 3031) as indicated in the figure by “X”.

The camera device 3000 includes a first optical chain includingnon-circular lens 3001 which has an oval aperture, a light redirectionelement 3003, a light path 3005 and a sensor 3007. Light enters thefirst optical chain and traverses a first portion of the first lightpath 3005 before impinging on the mirror 3003 and being redirected 90degrees or approximately 90 degrees towards the sensor 3007.

The camera device 3000 further includes a second optical chain includinga second non-circular lens 3011 which has an oval aperture, a secondlight redirection element 3013, a second light path 3015 and a secondsensor 3017. Light enters the second optical chain and traverses a firstportion of the second light path 3015 before impinging on the mirror3013 and being redirected 90 degrees or approximately 90 degrees towardsthe second sensor 3017.

The camera device 3000 further includes a third optical chain includinga third non-circular lens 3021 which has an oval aperture, a third lightredirection element 3023, a third light path 3025 and a third sensor3027. Light enters the third optical chain and traverses a first portionof the third light path 3025 before impinging on the mirror 3023 andbeing redirected 90 degrees or approximately 90 degrees towards thethird sensor 3027.

The camera device 3000 further includes a fourth optical chain includinga fourth non-circular lens 3031 which has an oval aperture, a fourthlight redirection element 3033, a fourth light path 3035 and a fourthsensor 3037. Light enters the fourth optical chain and traverses a firstportion of the fourth light path 3035 before impinging on the fourthmirror 3033 and being redirected 90 degrees or approximately 90 degreestowards the fourth sensor 3027.

Note that while the four lenses and corresponding apertures 3001, 3011,3021, 3031 are located on the front face of the camera 3000, they areintentionally positioned so that the longest extent of each aperture isat a different angle relative to the bottom of the camera housing whichduring use will normally coincide with the horizontal direction.

FIG. 30 shows how multiple lenses 3001, 3011, 3021, 3031 withnon-circular apertures can be used in a single exemplary camera device3000 to collect high frequency information in multiple directions sothat high frequency information, and thus high resolution, is availablein each of a plurality of directions when combining images to generate acomposite image. While the camera modules shown in FIG. 30 are shownutilizing a large portion of the space within the camera device fortheir individual light paths with the light paths crossing, it should beappreciated that the optical chains can, and in some embodiments are,implemented as individual self contained modules which have light pathswhich do not cross.

Such sealed self contained light modules can be particularly desirablein some embodiments where dust and/or other concerns over contaminationmay be an issue such as in camera embodiments intended for outdoorand/or beach use where dust and/or sand are of a concern.

FIG. 31 shows an exemplary scene 3102 including a scene area 3104 whichmay have all or portions of its image captured by camera modules of acamera implemented in accordance with one or more embodiments of theinvention. The exemplary scene 3102 includes a view of mountains 3112,birds 3113, a person 3106 and a house 3108. While the scene 3102 coversa very large area the scene area of interest 3104 includes the person3106 and house 3108. It should be pointed out that while reference ismade to a ‘scene’ or ‘scene area’ such references are not to beinterpreted as the physical scene which is 3 dimensional, but it shouldbe interpreted as the 2 dimensional projection or representation of thephysical scene obtained by capturing an image of the scene using anideal camera. Any reference to the area of a scene is to be interpretedas the area in such a 2 dimensional projection. In such a projectioneven a small object that is sufficiently close to the camera can appearto have a large area given its proximity to the camera while a distantobject will appear smaller than an equally sized object closer to thecamera.

FIG. 32 is a drawing 3200 showing how different optical chains, e.g.,camera modules, of a camera, such as the camera device 600 of FIG. 6 orthe camera device 1400 of FIG. 14, in which each camera device includesmultiple optical chains (as shown in FIGS. 12A and 17A), some of whichhave different focal lengths, can capture different size portions of ascene area of interest 3202 (which may correspond to scene area ofinterest 3104 shown in FIG. 31).

For purposes of discussion, the capture and combining of imagescorresponding to different scene areas will be explained using thecamera device 600 by referring to FIG. 12A which shows the arrangementof optical chains in camera 600.

Consider for purposes of discussion that the camera device 600 includesthe 17 modules arranged as shown in FIG. 12A. As previously discussed inthe FIG. 12A example, three different focal lengths, f1, f2 and f3 areused where f1<f2<f3; f1 is ½ f2; and f2 is ½ f3.

For purposes of discussion the first through seventh camera modules1202, 1206, 1210, 1212, 1216 1220, 1222, respectively, are the moduleswith the largest lenses (and thus largest apertures in variousembodiments) and largest supported focal lengths (f3). For simplicity inthe discussion below, it is further assumed that the distances betweenthe various camera modules is much smaller than the distance between thecamera and all the objects in the scene. This is however not alimitation of the described invention but meant only to make theexplanation easier to follow.

The five medium sized camera modules which are the eighth through 12thcamera modules correspond to reference numbers 1204, 1208, 1214, 1218,1224, respectively and have medium diameter lenses and medium supportedfocal lengths (f2).

The five camera modules which are the 13th through 17th camera modulescorrespond to reference numbers 1226, 1228, 1230, 1230 and 1234 and havethe smallest diameter outer lenses and smallest focal length (f1).

It should be appreciated that the camera modules with the largest focallength f3 will capture the smallest portion of a scene area of interestgiven that they provide the greatest magnification. Assuming that cameramodules of the different focal lengths use sensors with the same totalpixel count, the modules with the larger focal length (f3) will providean image with a higher pixel to scene area ratio since more pixels willbe used to capture an image of a smaller scene area than will be thecase with the medium (f2) and small focal length (f1) camera modules.

It should be appreciated that given the difference in magnificationbetween the modules with different focal lengths (f1, f2, f3) the scenearea captured by the small focal length (f1) camera modules willcorrespond to portion of the scene area of interest which isapproximately 16 times the size of the portion the scene area ofinterest which is captured by the camera modules with the largest (f3)focal length. The portion of the scene area of interest captured bycamera modules with the intermediate focal length (f2) will be 4 timesthe size of the portion of the scene area of interest captured by thecamera modules with the largest focal length (f3) and ¼ the size of theportion of the scene area of interest captured by the camera moduleswith the smallest focal length (f1).

The relationship between the scene areas captured by camera modulescorresponding to the f1 and f2 focal lengths can be appreciated in thecontext of the FIG. 32 example which shows 7 distinct scene areas.

In the FIG. 32 example scene area of interest is identified by reference3202. diagonals 3250 and 3260 are shown so that the relationship betweenthe different areas and how they correspond to each other can be betterappreciated. The first scene area 3204 and fourth scene area 3210 are ofsimilar size or are of the same size and correspond to the full scenearea of interest 3202 which corresponds to the exemplary scene area ofinterest 3102 shown in FIG. 31. For purposes of explanation considerthat the first and fourth scene areas are captured by optical chainshaving the focal length f1, i.e., by smaller focal length opticalchains. Assume for discussion purposes that (f1) camera module 1228 isused to capture the first scene area 3204 and that (f1) camera module1232 is used to capture the fourth scene area 3210. Note that the actualimage captured by 1228 and 1232 may be of a slightly larger scene areato ensure that the scene area of interest is captured.

We will also assume that f2 camera module 1204 is used to capture thesecond scene area, that (f2) camera module 1208 is used to capture thethird scene area, that (f2) camera module 1218 is used to capture thefifth scene area, that (f2) camera module 1214 is used to capture thesixth scene area 3214 and that (f2) camera module 1224 is used tocapture the seventh scene area 3216. Again as with the capture of theother scene areas, the actual images captured by the modules 1204, 1208,1218, 1214 and 1224 may be of slightly larger scene areas to ensure thatthe respective second 3206, third 3208, fifth 3212, sixth 3214 andseventh 3216 scene areas are fully contained in the captured images.

Note that the relative position of the outer lenses of the cameramodules shown in drawing 1200 are known and fixed. However, in someembodiments the modules 1204, 1208, 1218, 1214 and 1224 are the same orsimilar in there elements and function to the module 500 in FIG. 5 whichincludes a mirror 510 that can be driven, e.g., moved or rotated by thehinge (mirror) drive 516 to change the angle of the mirror 510. Whilethe mirror drive 516 can rotate the mirror around the hinge axis andthus change its angle, the hinge 508 prevents motion in other directionsand thus the optical axis (outside the camera) rotates in a planeperpendicular to the axis of the hinge. When the mirror 510 is at a 45degree angle, the light entering the lens 512 along it's optical axis isdeflected 90 degrees into the optical axis of Part B of the module 500.While we describe here a mirror 510 that is hinged and can rotate alongan axis, in some other embodiments the place of the mirror is moved to adifferent plane such that this motion is not constrained to be rotationalong any fixed axis. In this case the optical axis of the camera modulecan be made to point in any desired direction (towards any point in thescene of interest).

While some modules use mirror that are movable and hinged, in otherembodiments one or more of the camera modules are implemented with fixedposition mirrors allowing the moveable hinge 508 and mirror drive 516 tobe omitted. For example, in one embodiment the camera modules used tocapture the full scene area of interest have fixed mirrors while thecamera modules used to capture small portions of the scene area ofinterest each include a movably hinged mirror. While combinations ofcamera modules with some having fixed mirrors and others having movablemirrors can be used, in at least one embodiment each of the multiplecamera modules included in an exemplary camera device have movablemirrors.

The mirror/hinge drive 516 is controlled by the processor 110 dependingon the particular mode of camera operation. Thus, when a user selects afirst mode of operation one or more camera modules may have theirmirrors at a first angle while during another mode of operation, e.g., amodule in which images are to captured and combined as shown in FIG. 34,one or more camera modules will have their mirror driven to a differentposition under control of the processor 110. The particular mode ofcamera device operation may be determined based on user input by theprocessor 110 operating under control of the mode control module 111 ordirectly by the mode control module 111 when the mode control module isimplemented in hardware.

If mirrors in each of 1204, 1208, 1218, 1214 and 1224 are at 45 degrees,each module looks directly out of the front face of the camera and theiroptical axes are all parallel. In this case each of the modules willtake an image of essentially the same scene area, the Seventh Scene Areaof FIG. 32. To capture an image of the Second Scene Area with module1204, the hinged mirror 510 of module 1204 needs to be adjusted so thatthe optical axis of camera module 1204 points towards the center of thesecond scene area 3206. Note that the module 1204 is positioned in thecamera 1200 in such a manner that as the mirror rotates around thehinge, the location in the scene area of interest 3202 that the opticalaxis points to moves along the diagonal 3250 of 3202. Similarly, themirror for camera module 1214 needs to be adjusted to capture the SixthScene Area. Note that in FIG. 12, camera modules 1204, 1214 are arrangedproximate, e.g., along or adjacent, the diagonal 3250 while cameramodules 1208, 1218 are located proximate, e.g., along or adjacent, thediagonal 3260. Rotating the mirror in 1214, e.g., changing the angle andthus incline of the mirror, makes the module's optical axis move alongthe diagonal 3250. Mirrors of modules 1208 and 1218 are similarlyangled, e.g., rotated, to capture images of the Third (3208) and Fifth(3212) Scene Areas respectively. In the case of modules 1208, 1218 theoptical axes move along diagonal 3260 of the scene area of interest. Themodule 1224 used to capture the seventh image area 3216 points at thecenter of the scene area of interest 3202 so it's mirror is maintainedat 45 degrees.

It should be appreciated from the above discussion that it isparticularly beneficial to have at least some camera modules arrangedalong diagonals 3250 and 3260. These modules have the Part B of theiroptical axis parallel to one of these two diagonals. Thus, thearrangement of modules 1210, 1220, 2202, 1212 with the largest aperturesalong diagonals and also the arrangement of medium aperture modules1204, 1214, 1208, 1208 along the same diagonals but offset from theother modules for space reasons, is an intentional design choice becauseit facilitates image capture and combining in some embodiments and modesof operation.

Based on the overlapping scene areas, e.g., 3210 and 3204 a depth map isgenerated, e.g., by the processor included in the camera. In someembodiments the depth of an object in the scene can be determined byexamining the relative positions of an object in the images captured bydifferent modules. In at least some embodiments the depth map is used,e.g., in combination with information about the relative position of theouter lenses of the different optical chains and/or optical axis of theoptical chains in combining images captured by the different opticalchains to form a composite image. The use of the depth information inthe generation of the composite image allows for the correction ofparallax, perspective and/or other image distortions that may occur orwhich are present in the images.

In the FIG. 32 example, 7 distinct scene areas are shown for purposes ofexplaining the invention. Each of the 7 scene areas may be, and in someembodiments is, captured by a different optical chain of the cameradevice 600 shown in drawing 1200 prior to being combined. The cameramodules, as will be discussed below, can capture images at the sametime, e.g., in parallel. However, in some embodiments as will bediscussed below where rolling shutters are used the camera modules arecontrolled to capture portions of the scene area of interest in asynchronized manner so that all the different camera modules whichcapture a given portion of a scene area of interest will capture thegiven portion at the same time.

It should be appreciated that by combing seven images corresponding tothe seven different scene area portions shown in FIG. 32 to generate acomposite image, it is possible to generate a composite image with fourtimes the pixel count of a single image sensor. For example, if each ofthe image portions is captured by a camera module using an 8 mega pixelsensor, the composite image corresponding to the scene area of interestshown in FIG. 32 would have an overall pixel count of 32 megapixelssince the second, third, fifth and sixth scene area would each becaptured by a different 8 megapixel sensor and thus contribute 8megapixels to the composite image. The actual resolution could beslightly lower if the captured images are slightly larger than thecorresponding scene areas.

While the sensors used to capture the first and fourth scene areas arenot likely to result in an increase in the overall pixel count of thecomposite image since they correspond to the same image area as thatcaptured by the combination of sensors used to capture the second,third, fifth and sixth scene areas, they provide for increased lightcapture than would be possible without the use of the f1 lenses and alsoprovide important information which allows for the generation a depthmap and which provide images of the overall scene area which can be usedin aligning and stitching together the images corresponding to thesecond, third, fifth and sixth scene areas as part of the process ofgenerating the composite image.

The (f3) camera module, e.g., 1216, is used to capture the seventh scenearea. The center of the seventh scene area coincides with the center ofthe image area of interest. Since practically most lenses have the leastaberrations and best image quality at the center of their field of view,this ensures that the center of the scene area of interest is imaged athigh quality by the camera module capturing the seventh scene area. Theimaging of the seventh scene area also increases the total amount oflight energy captured at the center of the scene area of interest. Thisallows the composite image generated from the captured images to haveits best quality (high resolution and minimum noise) at the center ofthe scene area of interest.

It should be appreciated that images captured by single lens camera areoften better at the center than at the edges due to optical aberrationsand vignetting being greater near the edge of the field than at thecenter. The composite image generated in accordance with some embodimentwill show a similar high quality at the center of the image with thepossibility of lower quality towards the edges of the composite image.Given that this effect is similar to that of conventional single lenscameras, it should be appreciated that the composite images will besimilar to but potentially of higher quality than images captured by asingle camera module.

FIG. 33 shows how the different image portions captured in the FIG. 32example may relate to the exemplary scene area of interest shown in FIG.31 to give a better understanding of the invention. Each of the imageportions of the scene area of interest captured by different modules, asdiscussed in FIG. 32, are identified in FIG. 33 with the same referencenumber followed by a prime (′). For example 3202′ shows the fullcaptured scene area of interest that includes the scene of interest,3206′ is the second scene area and relates to one portion of the scenearea of interest, 3208′ is the third scene area and relates to anotherportion of the scene area of interest, 3210′ is the fourth scene areaand includes the full scene area of interest, 3212′ is the fifth scenearea and relates to another portion of the scene area of interest, 3214′is the sixth scene area and relates to another portion of the scene areaof interest and 3216′ is the seventh scene area and relates to thecenter portion of the scene area of interest. How each of the capturedimage portions relate to the exemplary scene area of interest shown inFIG. 31 can be better appreciated from FIG. 33.

FIG. 34 is a drawing 3400 showing different image portions captured by acamera having optical chains, e.g., camera modules, corresponding tothree different focal lengths, e.g., (f1, f2 and f3). As discussed invarious sections above, in some embodiments each of the optical chainsin a camera includes a corresponding image sensor that captures lightwhen an image portion is captured by the corresponding optical chain. Inone embodiment a first optical chain including a first sensor captures afirst image of a scene of interest 3404 (shown as a medium sizerectangle 3404). A second optical chain including a second sensorcaptures an image portion 3406 (shown as a small size rectangle 3406) ofthe scene of interest, a third optical chain including a third sensorcaptures another image portion 3408 of the scene of interest, a fifthoptical chain including a fifth sensor captures another image portion3410 of the scene of interest, a sixth optical chain including a sixthsensor captures another image portion 3412 of the scene of interest anda seventh optical chain including a seventh sensor captures anotherimage portion 3414 of the scene of interest. Furthermore, a fourthoptical chain including a fourth sensor captures an image 3402 thatincludes the entire scene of interest. In one embodiments the focallength of the second, third, fifth, sixth, and seventh optical chains isthe same (f3) and is greater than the focal length (f2) of the firstoptical chain that captures scene area of interest 3404 and the focallength (f1) the fourth optical chain that captures the scene area 3402.In one such embodiment the focal length of the various optical chainsare such that f1<f2<f3.

While one embodiment is discussed above, it should be appreciated thatvarious different variations are possible. In one such differentembodiment, the different image portions shown in FIG. 34 are capturedby a camera having optical chains corresponding to two different focallengths, e.g., (f1, f2), where image portions 3406, 3408, 3410, 3412,3414 are captured by optical chains having a focal length (e.g., f2)which is greater than a focal length (e.g., f1) of the optical chainthat captures image portions 3402 and 3404. In one such embodiment, f2is twice f1. FIG. 34 and the image portions shown therein will be usedin explaining how rolling shutters corresponding to different cameramodules can be controlled in a coordinated manner to facilitatecombining of images captured by different camera modules in a way thatreduces or minimize motion related (camera or subject related)distortions that may be introduced if each of the camera module sensorswere independently (asynchronously) operated to capture the imageportions. The read out from the sensors of the camera modules in acoordinated manner helps in minimizing distortions due to uncoordinatedasynchronous image capturing by different optical chains and thecaptured images can be combined easily.

The above discussed image capture operations performed by varioussensors included in corresponding optical chains as discussed above may,and in some embodiments is, performed by a camera such as camera 1400including optical chains arranged as illustrated in FIG. 17. In anotherembodiment the image capture operations performed by the sensorsincluded in corresponding optical chains as discussed above is performedby the camera 600 including optical chains arranged as illustrated inFIG. 12.

FIG. 35 shows how sensors of four optical chains, e.g., camera modules,using rolling shutters can be used in combination to capture a scenearea of interest in a manner that facilitates combining of the capturedimages with one or more images captured by another optical chain havinga smaller focal length which captures a larger portion of the same scenearea of interest. Drawings 706, 708, 710 and 712 illustrate howexemplary image sensors corresponding different optical chains arecontrolled to perform a read out from one edge to another (e.g., top tobottom) as part of image capture operation. For example, drawing 706shows the second image sensor discussed with regard to FIG. 34 that hasN rows of pixel elements that can be read out, e.g., from top to bottom,as part of capturing the image portion 3406 (shown in FIG. 34) by asecond optical chain to which the image sensor shown in 706 corresponds.Drawing 708 shows a third image sensor, discussed above with regard toFIG. 34, that has N rows of pixel elements that can be read out from topto bottom as part of capturing image portion 3408 by the third opticalchain to which the image sensor shown in 708 corresponds. Similarly 710and 712 show fifth and sixth sensors each with N rows of pixel elements,capturing image portions 3410 and 3412 shown in FIG. 34. The read outfrom the sensors shown in drawings 706, 708 starts synchronously at thesame time and read out from the sensors shown in drawings 710, 712starts together when the read out from sensors shown in drawings 710,712 ends.

FIG. 36 shows how a sensor corresponding to a optical chain, e.g., firstoptical chain discussed with regard to FIG. 34, having a focal lengthhalf or approximately half that of the optical chains (second, third,fifth and sixth optical chains discussed with regard to FIG. 34) used tocapture the image portions 3406, 3408, 3410 and 3412 shown in FIG. 34,can be controlled so that the optical chain with the smaller focallength captures all or portions of scene of interest in a synchronizedmanner with multiple optical chains having a larger focal length.Drawing 3600 shows the first sensor with N rows of pixel elements thatcan be read out in a manner synchronized with the read out of pixel rowscorresponding to other sensors corresponding to optical chains withlarger focal length shown in FIG. 35. While drawing 3600 illustratesthat the first sensor corresponding to the relatively smaller focallength has N rows of pixel elements, it should be appreciated that therectangle in drawing 3600 does not show or represent the physical sizeof the first sensor.

FIGS. 35, 36, 37, 38 and 39 show various aspects relating to rollingshutter control of the reads of sensors of different camera modules in acoordinated manner so that the images captured by the different sensorscan be easily combined. As should be appreciated in embodiments whererolling shutters are used rows of pixel values are read outsequentially. In the FIGS. 35, 36, 37, 38 and 39, it is assumed thateach sensor includes the same number of rows of pixel elements, e.g., Nrows. Accordingly, for the purposes of discussing these figures, N isthe number of rows of pixel values on a sensor and to facilitate adiscussion of the rolling shutter control it is assumed that the sensorof different camera modules have the same number of rows of pixelelements. While the sensors have N rows of pixel elements, in someembodiments a joint read of rows is permitted in which case the sensoroperates as if it has N/M rows of pixel elements where M is the jointread factor, e.g., the number of rows which are read out jointly, e.g.,summed and read out, thereby producing the same number of pixel valuesas the read out of a single row of pixel elements. A joint readoperation has the advantage that a full read out of the sensor can becompleted in 1/Mth the time it would take to read out the sensor ifindividual rows are read separately.

It should be appreciated that in case of an electronic rolling shutterimplementation, the rows of the image sensor are read out in sequence.The time taken to read each row is quite small but the number of rowscan be large and the minimum time taken to read the entire image can beas much 50 milliseconds or more. Since the row read time is small, weshall assume that all the pixels in a given row are read (nearly)simultaneously. The integration of light for a given row of pixelsbegins when the row is reset (opening curtain) and ends when the row isread out (closing curtain). The exposure duration is the durationbetween the row reset time and row reading time. The exposure instant or(capture time) of a given row can be defined as the time at which thatrow is read out. More precisely the exposure instant may be consideredas the average of the row reset and row read times. Clearly scene areascorresponding to different rows of an image sensor are captured atdifferent times. When portions of a given scene area of interest arecaptured by multiple camera modules, each implementing an electronicrolling shutter, with the intention of combining the multiple capturedimages to create one image, motion related artifacts can be observed ifthe rolling shutters of the various modules are not synchronized. Inorder to minimize these artifacts, the rolling shutters of the cameramodules are synchronized such that the images of any given point of thescene area are captured by the different sensors at (nearly) the sameinstant.

Assuming it takes a time period T_(R) to read a row of pixel values, tocomplete a full read of a sensor having N rows, it will take a timeperiod of N times T_(R). If multiple sensors corresponding to differentportions of the scene area are operated to capture the images as quicklyas possible, it is possible that combining the images would result inartifacts due to portions of a scene captured at different times beingcombined.

It should be appreciated that to avoid such temporal (motion related)artifacts it is desirable to have different sensors capture the sameportion of a scene area at the same time. It should also be appreciatedthat when combining images from multiple camera modules having a largefocal length with one or more images captured by camera modules with asmaller focal length (and thus which capture a larger part of a scene)it may be desirable for some of the larger focal length modules tocapture portions of the scene sequentially, e.g., as if a rollingshutter was being controlled across a larger sensor having a number ofrows which is equal to some combination of the number of rows of thesensors of the camera modules with the larger focal length and whichthus capture an image corresponding to the smaller portion of a scene.

FIG. 35 illustrates how the N rows of pixel elements shown in drawings706, 708, 710, 712 corresponding to four sensors each of whichcorresponds to a different camera module can be read out sequentially.Note that the rows of pixel elements corresponding to the four sensorsshown in drawings 706, 708, 710, 712 can capture four different portionsof a scene such as scene portions 3406, 3408, 3410 and 3412.

Assume for purposes of discussion that the rows of pixel elements shownin drawings 706, 708, 710, 712 correspond to sensors of the second,third, fifth and sixth optical chains each of which has a focal lengthf3. As will be discussed below, f3 is twice a focal length f2 of a firstoptical chain that captures scene of interest 3404 and four times afocal length f1 of a fourth optical chain that captures scene area 3402in some embodiments. Thus the optical chains using the focal length f3will provide relatively high magnification but capture a smaller portionof a scene than the optical chains using the focal length f2 or thefocal length f1. The optical chains using the focal length f3 willcapture an image of a scene area ¼ the size of the optical chains havingthe focal length f2 and 1/16 the size of optical chains having the focallength f1. While drawing 3500 illustrates that the sensors,corresponding to the larger focal length optical chains that captureportions of scene of interest, have N rows of pixel elements, it shouldbe appreciated that the rectangles in drawings 706, 708, 710, 712 do notshow or represent the physical size of the sensors but rather therelative size of the image area captured by the sensor.

This relationship of focal length to scene area capture size isrepresented in FIGS. 34, 35, 36 and 37 by the relative sizes of the rowsof pixel elements shown. Thus, it should be appreciated that while theactual size of the pixel elements may be the same, in each of thesensors, the size of the portion of the scene to which the capturedpixel values correspond depends on the focal length of the camera modulein which the particular sensor is included.

For purposes of discussion, it will be assumed that drawings 706, 708,710, 712 of FIG. 35 show rows of pixel elements corresponding to foursensors corresponding to optical chains having the focal length f3. Itis also assumed that the sensor corresponding to drawing 706approximately captures a top left portion of a scene of interest, thesensor corresponding to drawing 708 approximately captures a top rightportion of the scene or interest, the sensor corresponding to drawing710 approximately captures lower left portion of the scene of interestwhile the sensor corresponding to drawing 712 approximately captures alower right portion of the scene of interest.

Assuming the sensor corresponding to the drawing 706 is a second sensorS2, the readout of the first line of pixel row 1 will be completed bythe time indicated by RS₂L₁ which stands for Read Sensor 2 Line 1.Similar annotation is use to indicate the point at which the readout oflines of pixel elements of other sensors is completed with timecorresponding to the vertical axis of FIGS. 35-39 and increasing fromtop to bottom of the figure. The read out of Sensor 3 corresponding todrawing 708 will be completed at the same point in time as the readoutout of RS₂L₁. Note that in the FIG. 35 example, there is no overlapbetween the image areas captured by the four sensors S2, S3, S5, S6 withthe readout out of Sensor S5 being completed as if it corresponded tothe next pixel row following the last pixel row of sensor S2. It shouldbe noted that if sensors had rows of pixel values corresponding tooverlapping image portions they would be read out at the same time butthe FIG. 35 example assumes no overlap in captured image areas which isnot the case in many embodiments.

For purposes of discussion, it will be assumed that T_(R) is the time toread one row of pixel values from any one of the sensors used in theFIG. 35 example. Since each sensor in the example includes N rows, thetime required to fully read any one of the N sensors is N T_(R). Thus,the readout of row (line N) of the first and thirds sensors S2, S3 willbe completed at time NTR while the readout of the last row (line N) ofsensors S5 and S6 will be completed at time 2 N TR.

If the sensors included different numbers of rows of pixels the totalreadout time would of course depend on the number of rows in the sensor.

While it might seem desirable to read out the sensors in the same timeperiod N T_(R), when the images from the different sensors are beingread out using a rolling shutter it may be desirable to control thesensors so that pixel rows corresponding to sensors shown in drawings710, 712 are read out following a read out of pixel rows correspondingto sensors shown in drawings 706, 708. In this manner, the read out willoccur sequentially as if the pixel rows corresponding to drawings 706,710 were part of a single rolling shutter. This approach which is usedin some embodiments allows for the composite image to be similar to whatwould have been obtained using a single larger sensor with 2 N rows ofpixel elements rather than two separate sensors each with N rows.Avoiding the temporal discontinuity that might occur between the lastrow of one sensor and the first row of the next sensor can thus haveadvantages when combining images captured by sensors corresponding todifferent portions of an image.

Thus, it should be appreciated that the sensors of the four cameramodules using rolling shutters corresponding to row of pixel elementsshown in drawing 706, 708, 710, 712 can be used in combination tocapture a scene area of interest in a manner that facilitates combiningof the captured images.

The captured images can be combined together to form a composite imageof a larger scene area and with one or more images of the larger scenearea. The images of the larger scene area can, and in some embodimentsare captured by camera modules having a smaller focal length, e.g., afocal length f2, which captures a larger portion of the same scene.

FIG. 36 shows how a sensor corresponding to an optical chain, e.g.,camera module, having a focal length f2, which is half or approximatelyhalf the focal length f3 of the camera modules used to capture sceneareas represented by the sets of rows of pixel elements shown in FIG.35, can be controlled so that the optical chain with the smaller focal(f2) length captures portions of scene of interest in a synchronizedmanner with multiple optical chains having a longer focal length. InFIG. 36, the rate at which rows of pixel values are read out is reducedby a factor which is based on the relative ratio of the focal length toone or more other supported focal lengths.

Since focal length f2 is ½ that of f3, the sensor corresponding todrawing 3600 including N pixel element rows capture substantially thesame image portions of the scene that are captured by the sensors of theFIG. 35 example, the read out rate of the pixel rows is reduced by afactor of 2, that is, one pixel elements row will be read out in timeperiod 2T_(R). Thus the read out of the sensor corresponding to drawing3600 that includes N pixel element rows will take a total time period ofN(2T_(R)) to complete a full read out of the pixel rows which is thesame amount of time that will be used by the combination of the sensorscorresponding to drawings 706, 710 to be fully read out. Assuming FIG.36 corresponds to a sensor S1, the readout of the first row of pixelvalues of S1 will be completed at time 2TR which corresponds to thepoint in time indicated by the indicator RS1L1 which stands for ReadSensor 1 Line 1. Thus, in such an embodiment where the read out of thepixel rows of FIGS. 35 and 36 are synchronized, the sensor S1 havingpixel element rows shown in drawing 3600 will begin its read out at thesame time the sensor S2 corresponding to drawing 706 and Sensor S3 whichcorresponds to drawing 708 begin their read outs of their first rows ofpixel elements. Sensor S1 will complete the read out of its N rows ofpixel elements at or near the same time the last row of pixel elementsshown in drawings 710 and 712 are read out. Thus, sensor S1 will beginits readout of pixel rows and end its pixel row readout in a manner thatis synchronized with the sensors to which FIG. 35 corresponds.

While the readout of sensor S1 could have been completed in time NT_(R)if the sensor was read out at the same per row read out rate as thesensors S2, S3, S5, S6, this would result in the undesirable effect ofsensors corresponding to different optical chains capturing the sameportion of an image area at different times which can lead to temporalartifacts in a composite image generated from images including an objectmoving at a high rate of motion. Such temporal artifacts are reduced oravoided by the synchronized image capture technique described herein.

FIG. 37 shows how a sensor, S4, corresponding to an optical chain (e.g.,fourth optical chain discussed with regard to FIG. 34) having a focallength (e.g., f1) one fourth or approximately one fourth the focallength f3 of the camera modules used to capture the images shown in FIG.35 can be controlled so that the corresponding optical chain capturesportion of scene of interest in a synchronized manner with multiplecamera modules having larger focal lengths.

Since focal length (e.g., f1) of the optical chain to which the sensorof drawing 3700 corresponds is ¼ that of f3, the sensor of drawing 3700including N pixel element rows capture substantially the same imageportions of the scene that are captured by the sensors of the FIGS. 35and 36 example, the read out rate of the N pixel rows of the sensor ofdrawing 3700 is reduced by a factor of 4, that is, one pixel elementsrow will be read out in time period 4T_(R). In FIG. 37 drawing 3700shows the sensor having N pixel element rows which is controlled to readout one row of pixel values every time period having a duration 4 T_(R)taking into consideration that the sensor covers a scene portion whichis 4 times larger in the vertical dimension than the sensorcorresponding to drawing 706 of FIG. 35. While drawing 3700 illustratesthat the sensor corresponding to the optical chain with small focallength f1 has N rows of pixel elements, it should be appreciated thatthe rectangle in drawing 3700 does not show or represent the physicalsize of the sensor. The size of the sensors (including pixel elements)in the various optical chains may be the same.

As should be appreciated, the read out of the first line of sensor S4will be completed at time RS4L1 with the Nth line being read out at timeRS4LN. Note that given that the readout rate of a row of pixel valuesoccurs once every 4 TR in FIG. 37 to maintain synchronization with thesensor readouts shown in FIGS. 35 and 36, the readout of sensor S4 willoccur at time 4NT_(R) assuming a start time of 0.

FIG. 38 shows the row readout timing relationships between the sensorsused in the FIGS. 35, 36 and 37 embodiments in greater detail when theyare used together in a synchronized manner. For illustration anddiscussion purposes, the same image and/or image portions captured byvarious optical chains as shown in FIG. 34 and discussed in thecorresponding description above, have been used in FIGS. 38-39 forreference. Accordingly, the same reference numbers have been used. Block3402 represents a scene area captured by a sensor corresponding to aoptical chain with focal length f1. Block 3404 represents portion of thescene area 3402 which is captured by a sensor with focal length f2.Blocks 3406, 3410, 3408, 3412 and 3414 represent blocks of a scene areacaptured by different camera modules having a focal length f3 whichprovides the highest magnification in the FIG. 38 example with thesmallest portion of the scene area being captured.

FIG. 39 shows a particular example similar to that shown in FIG. 38 butwhere the number of rows N is selected to be 6, that is, each of thesensor includes 6 rows of pixel elements, e.g., N=6, with theunderstanding that the number of rows in most sensors will be muchhigher. Thus, FIG. 39 is shown for exemplary purposes to illustrate howsynchronization may be achieved for a simple case where N=6 and eachsensor in the example is of the same size but corresponding to opticalchains with different focal lengths.

It should be appreciated from FIG. 39 that when sensors capture imagescorresponding to the same portions of a scene area, the readouts willoccur in parallel. Once a line of pixel values is read out, the nextreadout will not occur until the synchronization read out controlindicates that the next row of unread pixel values is to be read out.

When a sensor corresponds to an image area for which there is no othersensor, the sensor will read out its row of pixel values while the othersensors refrain from performing a read out operation. For example, inthe FIG. 39 example, sensor S4 which has the smallest focal length andcaptures the largest image area will read out its first row of pixelvalues while sensors S1, S2, S3 and S6 refrain from reading out pixelvalues. However, when a sensor captures a portion of an image area whichis also captured by other sensors it will read out the pixel values in amanner that is synchronized to closely coincide with the readout of thesensor or sensors capturing the same image area. For example, during thetime period extending from Time RS3L3 to time RS4L4, sensor S1, SensorS2 and Sensor S3 will also perform pixel readouts. The number ofreadouts performed during this time period will depend on the rowreadout rate of the sensor with sensors corresponding to optical chainshave a larger focal length reading out multiple rows of pixel values inthe time period in which sensor S4 reads out a single row of pixelvalues.

Note that T_(R) which is a unit of time indicating the minimum amount oftime used to read a row of pixel elements may be expressed inmilliseconds or any other suitable unit of time. Thus, the vertical axisis intended to show time progressing from the top to the bottom based onassumption that the scan of the scene area and read out of the sensorsstarts at the top and proceeds towards the bottom. While a top to bottomscan direction is shown for purposes of the example, this is notcritical or important to the timing synchronization features.

It should be appreciated that while the method of synchronizing readoutrates based on focal lengths of the optical chain to which a sensorcorresponds have been explained in the simple case where the sensors areof the same size and include the same number of rows, the methods arewell suited for use with optical chains which use different size sensorsand/or sensors with different numbers of rows of pixel values. As shouldbe appreciated such differences, if present can, and in some embodimentsare, taken into consideration as to the start and stop times as well atthe readout rate to be used to control the readout of the sensors in amanner which will achieve synchronized image capture of an image area ofinterest by multiple optical chains, e.g., camera modules. Furthermoreoverlap in the image areas captured by different modules can be takeninto consideration when determining the start times for reading out ofdifferent sensors which may correspond to partially overlapping imageareas.

For example if sensor S5 of the FIG. 35 example captured an image areawhich overlapped image portions captured by some of the last rows ofsensor S2, the readout of the first row of S5 pixel elements would occurin parallel with the row of pixel elements of sensor S2 which capturedthe same image portion. In such an embodiment the readout of some butnot all of the rows of pixel elements of S5 would follow the completionof the read out of sensor S2 due to the partial overlap in image areaswhich were being captured by S2 and S5.

While FIGS. 36 to 39 have been used to explain how the read out rate ofsensors corresponding to camera modules with small focal lengths can beslowed down to synchronize them to the image capture and pixel read outof sensors corresponding to camera modules with larger focal lengths,where maximizing pixel count in an image is not of top priority, jointreads of pixel rows can, and in some embodiments is, used to speed upthe read out of the sensors corresponding to the higher focal lengthcamera modules. During a joint read of a group of M rows, the sensoruses analog circuitry to sum the corresponding pixels of each of the Mrows before digitization and reading. A joint read of M rows typicallytakes the same time as reading a single row thereby speeding up thereading speed by a factor of M. Such an approach allows for synchronizedrolling shutter control capture of a scene area of interest in a timeperiod equal to the time period required to read out a single sensor.While such an approach may result in a lower overall image pixel countin a composite image, e.g., with the pixel count of the composite imagebeing reduced in some cases to the pixel count of a single sensor, suchan approach may be desirable in cases where motion of an object in theimage being captured is present since the motion may introduce artifactsif the overall image capture period is increased since the object willmove further in the capture time period than if the capture occurred ina shorter time period.

FIG. 40 is an exemplary method of capturing images using multipleoptical chains, e.g., camera modules, and combining the images inaccordance with one exemplary embodiment.

FIG. 40 shows a flowchart 4000 illustrating the steps of an exemplarymethod of controlling an imaging device, e.g., such as that shown inFIG. 6, 8 and/or 14, that includes at least one sensor with a rollingshutter to generate a composite image in accordance with an exemplaryembodiment. The camera device implementing the method of flowchart 4000can and sometimes does include the same or similar elements as thecamera device 100 of FIG. 1 and device 200 of FIG. 4A.

The method of flowchart 4000 can be, and in some embodiments is,performed using a camera device such as the camera 100 of FIG. 1. Insome embodiments the camera 100 is a battery operated handheld device.The exemplary method starts in step 4002, e.g., when a user of a cameradevice, e.g., camera 100, presses a button or takes another action totrigger the capture of an image of a scene area of interest. For thepurposes of discussion consider that the camera device includes aplurality of optical chains, e.g., camera modules, and each of thecamera modules can be independently operated and controlled. Forpurposes of discussion, the capture and combining of imagescorresponding to different scene areas will be explained by referring tothe optical chains illustrated in FIG. 12A and using that as the basisof the example. As previously discussed in FIG. 12A example, threedifferent focal lengths, f1, f2 and f3 are used where f1<f2<f3.

Operation proceeds from step 4002 to steps 4004, 4006, 4008, 4010, 4012,4014 and 4016 which involve image capture operations. The image captureoperations may and in some embodiments are performed in a synchronizedmanner. In at least some synchronized embodiments the images captured bysome by not necessarily all of the different optical chains correspondto the same or an overlapping time period. In other embodiments imagecapture is not synchronized but multiple one of the captured images arecaptured during the same or an overlapping time period. In still otherembodiments as least some images are captured sequentially, e.g., inrapid succession. Sequential image capture may, and in some embodimentsare used for capturing images corresponding to different portions of ascene area. In step 4004 a first optical chain, e.g., a first cameramodule 1228, of the camera having a first focal length (f1) is operatedto capture a first image of a scene area of interest. In someembodiments the scene area of interest may be slightly smaller than thefull image capture area. Operation proceeds from step 4004 to step 4018.

In step 4006 a second image of a second scene area is captured using asecond optical chain of the camera, e.g., optical chain 1204, having asecond focal length (f2) which is greater than the first focal length(f1), the second scene area being a first portion of the scene area ofinterest. In various embodiments the second optical chain captures apart, e.g., quarter or half, portion of the scene area of interest.Operation proceeds from step 4006 to step 4018.

In step 4008 a third image of a third scene area is captured using athird optical chain of the camera, e.g., optical chain 1208, having athird focal length (f2) which is greater than the first focal length, atleast a portion of the third scene area being non-overlapping with thesecond scene area, the third scene area being a second portion of thescene area of interest. Thus in some embodiments the third optical chaincaptures a part, e.g., another quarter, portion of the scene area ofinterest. Operation proceeds from step 4008 to step 4018. In someembodiments the first, second and third optical chains have outer lenseson a front face of the camera. In some but not all embodiments, theouter most lens of the optical chain is a plane glass or plastic lenswith zero optical power.

In step 4010 a fourth image is captured using a fourth optical chain ofthe camera, e.g., optical chain 1234, having a fourth focal length whichis equal to or smaller than f1, the fourth image being of the scene areaof interest. In some embodiments the fourth focal length is the same orsmaller than the first focal length. Operation proceeds from step 4010to step 4018.

In step 4012 a fifth image of a fifth scene area is captured using afifth optical chain of the camera, e.g., optical chain 1218, having afifth focal length which is greater than the first focal length (f1),the fifth scene area being a third portion of the scene area ofinterest. Thus the fifth optical chain captures a part, e.g., quarter orhalf, portion of the scene area of interest. Operation proceeds fromstep 4012 to step 4018.

In step 4014 a sixth image of a sixth scene area is captured using asixth optical chain of the camera, e.g., optical chain 1214, having asixth focal length which is greater than the first focal length, thesixth scene area being a fourth portion of the scene area of interest.Operation proceeds from step 4014 to step 4018.

In step 4016 a seventh image of a seventh scene area is captured using aseventh optical chain of the camera, e.g., optical chain 1224, having aseventh focal length which is greater than the first focal length, theseventh scene area being at the center of the scene area of interest.Operation proceeds from step 4016 to step 4018.

In some embodiments the first and fourth focal lengths are the same. Insome embodiments the second, third, fifth and sixth focal lengths arethe same. In some embodiments the seventh focal length is the same asthe second, third, fifth and sixth focal lengths. In some embodimentsthe union, e.g., combination, of the second, third, fifth and sixthscene areas includes the scene area of interest.

Returning now to step 4018. Step 4018 is performed in some but notnecessarily all embodiments. In step 4018 depth information, for aportion of the scene area of interest which is included in at least twoof the captured images, is generated. Thus in some embodiments togenerate the depth map information for a portion of the scene area ofinterest, at least two captured images of the same scene area ofinterest are used. In some embodiments step 4018 includes step 4020. Insuch embodiments at least the first and fourth captured images are usedto generate the depth information as shown in step 4020. While the firstand fourth captured images are used to generate the depth information inone embodiment, other captured images may also be used in generating thedepth information. In various embodiments the depth information isgenerated by a processor, e.g., such as the processor 110 or 211,included in the camera.

In various embodiments the second, third, fifth, sixth and seventhoptical chains are arranged in such a manner that the images of theportion of the scene taken by these optical chains are from differentspatially separated entrance pupils and thus have differentperspectives. Combining such images with different perspectivesintroduces artifacts, e.g., parallax. To minimize and/or alleviate theeffect of such artifacts from a composite image generated using thevarious images captured by these different optical chains, the depthinformation is used which provides for parallax correction when combingthe images to avoid distortions of the composite image due to thedifferent perspectives.

Operation proceeds from step 4018 to step 4022. In step 4022 a compositeimage is generated from at least the first, second and third capturedimages. In some embodiments the step of generating a composite imageincludes step 4024 where depth information is used in generating thecomposite image. In such embodiments the composite image is generatedfrom at least the first, second and third captured images as a functionof the generated depth information, e.g., depth information generated instep 4018. While depth information is used in generating a compositeimage in some embodiments, the use of depth information in not necessaryin all embodiments for generating a composite image. In some embodimentsthe step 4022 of generating the composite image further includes usingthe fourth, fifth, sixth and seventh captured images to generate thecomposite image as illustrated in step 4026 shown. The composite imageincludes an image of the scene area of interest. In various embodimentsthe generated composite image is an image of the scene area of interest.Operation proceeds from step 4022 to step 4028. In step 4028 one or morecaptured images, e.g., first, second, third, fourth, fifth, sixth and/orseventh image, and/or the composite image is stored, e.g., in a devicememory and/or output, e.g., to a display device and/or to an externaldevice via an interface.

In some exemplary embodiment an imaging device such as e.g., the cameradevice 100, is used to implement the method of flowcharts 4000. In onesuch embodiment the plurality of optical chains 130 of the camera device100 include optical chains arranged in the manner as illustrated in FIG.12A with more detailed arrangements and elements (e.g., sensors,filters) of the optical chains further shown in FIG. 12B. Thus in suchan embodiment the plurality of optical chains 130 include optical chains1202 through 1234 discussed with regard to FIG. 12A. In anotherembodiment the plurality of optical chains 130 of the camera device 100include optical chains of the type and arrangement as illustrated inFIG. 17A. In such an embodiment the plurality of optical chains 130include optical chains 1702 through 1734 discussed with regard to FIGS.17A and 17B.

In some exemplary embodiments the processor 110 is configured togenerate a composite image by combining two or more images captured bythe optical chains as discussed above. In one embodiment the processor110 is configure to generate a composite image from at least a firstimage of a scene area of interest captured by the first optical chain, asecond image of a second scene area which is a first portion of thescene area of interest captured using the second optical chain, and athird image of a third scene area captured using the third opticalchain, the third scene area being a second portion of the scene area ofinterest, at least a portion of the third scene area beingnon-overlapping with the second scene area.

In some embodiments the processor 110 is configured to generate depthinformation for a portion of the scene area of interest which isincluded in at least two of the captured images. In some embodiments theprocessor 110 is further configured to generate the depth informationusing at least the first image captured by the first optical chain and afourth captured image captured by the fourth optical chain, the fourthimage being of the scene area of interest. In some such embodiments theprocessor 110 is further configured to generate the composite image as afunction of said depth information. In some embodiments the fourthoptical chain has a fourth focal length which is equal to or smallerthan the first focal length of the first optical chain. In someembodiments the processor 110 is further configured to control storageof the one or more captured images and/or the generated composite imagein the memory 108 and/or output of the one or more captured imagesand/or composite image on the display 102 and/or transmission of thecaptured images or the composite image to another device via aninterface such as interface 114.

It should be appreciated that various features and/or steps of method4000 relate to improvements in cameras and/or image processing eventhough such devices may use general purpose processors and/or imagesensors. While one or more steps of the method 4000, such as the stepsof generating depth information and generating composite image, havebeen discussed as being performed by a processor, e.g., processor 110,211, it should be appreciated that one or more of the steps of themethod 4000 may be, and in some embodiments are, implemented bydedicated circuitry, e.g., ASICs, FPGAs and/or other applicationspecific circuits which improve the efficiency, accuracy and/oroperational capability of the imaging device performing the method. Insome embodiments, dedicated hardware, e.g., circuitry, and/or thecombination of dedicated hardware and software are utilized inimplementing one or more steps of the method 4000 therein providingadditional image processing efficiency, accuracy and/or operationalcapability to the imaging device, e.g., camera, implementing the method.

FIG. 41 which comprises the combination of FIGS. 41A and 41B, is aflowchart 4100 illustrating the steps of an exemplary method ofoperating an imaging device, e.g., such as that shown in FIG. 6, 8and/or 14, that includes at least one image sensor with a rollingshutter to scan a portion of a scene area of interest in accordance withan exemplary embodiment. The camera device implementing the method offlowchart 4100 can and sometimes does include the same or similarelements as the camera device of FIG. 1 and FIG. 4A.

The exemplary method starts in step 4102, e.g., with a user initiatingthe capture of a scene area of interest on a camera device which causesthe camera device, e.g., camera device 100, to initiate a scan and thusimage capture of the scene area of interest by one or more opticalchains. For the purposes of discussion consider that the camera deviceincludes a plurality of image sensors wherein each sensor is a part ofan optical chain, e.g., first through X^(th) sensors being included in afirst through X^(th) optical chain respectively. Operation proceeds fromstep 4102 to step 4104. In step 4104 a plurality of image sensorsincluded in the camera are controlled to scan the scene area of interestfrom a first edge of the scene area of interest to a second edge of thescene area of interest, the first (e.g., top) and second (e.g., bottom)edges of the scene area of interest being parallel, the scanningincluding controlling image capture according to a scan position whichprogresses from the first edge to the second edge of the scene area ofinterest (e.g., at a uniform rate). In some embodiments the plurality ofimage sensors include multiple image sensors which capture substantiallynon-overlapping portions of the scene area of interest (in someembodiments the majority of image portions do not overlap) and the unionof the portions of the scene area of interest captured by the multiplesensors cover the scene area of interest. In some embodiments themultiple image sensors include an even number of sensors, e.g., 4.

In some embodiments the plurality of image sensors includes a first, asecond and a third image sensor, the first image sensor capturing afirst portion of the scene area of interest, the second image sensorcapturing a second portion of the scene area of interest, and the thirdimage sensor capturing the entire scene area of interest. In some suchembodiments the first and second portions of the scene area of interestare partially non-overlapping.

In some embodiments the first sensor captures an image of a portion ofthe scene area of interest which is not captured by the second sensor,and the second sensor captures an image of a portion of the scene areaof interest which is not captured by the first sensor. In someembodiments a first portion of the scene area of interest captured bythe first sensor begins at a scan location which precedes a secondportion of the scene area of interest captured by the second sensor. Insome embodiments the second portion of the scene area of interestcorresponds to the center of the scene area of interest. In someembodiments plurality of sensors includes 5 sensors, four of whichcapture substantially non-overlapping portions of the scene area ofinterest. In some embodiments first through fourth different cornerportions of the second portion of the scene area of interest overlapdifferent corresponding portions of the scene area of interest which arecaptured by the remaining four sensors of the five sensors.

In some embodiments the plurality of sensors includes 5 sensors, four ofwhich capture four different portions of the scene area of interest, atleast one corner of each of the four different portions of the scenearea of interest not overlapping an image portion captured by the otherthree of the four different portions of the scene area of interest, thefifth sensor capturing a center portion of the scene area of interest.

In various embodiments step 4104 includes performing one or more of thesteps 4106, 4108 and 4110. As part of controlling one or more sensors toscan the scene area of interest, in step 4106 a first sensor in theplurality of sensors is operated in rolling shutter mode to read out anas yet unread row of pixel values corresponding to a current scanposition when the first sensor has an unread row of pixel valuescorresponding to the current scan position, the current scan positionchanging over time (e.g. as the scan progresses from the first edge tothe second). In some embodiments, the scan position progresses from thefirst edge of the scene area of interest to the second edge of the scenearea of interest at a uniform rate.

In step 4108 a second sensor in the plurality of sensors is operated inrolling shutter mode to read out an as yet unread row of pixel valuescorresponding to the current scan position when the second sensor has anunread row of pixel values corresponding to the current scan position,the first and second sensors capturing images of different portions ofthe scene area of interest (e.g. the different portions may be partiallyoverlapping or one may be inside the other). Similarly, as part ofcontrolling scanning the scene area of interest, in step 4110 an X^(th)sensor in the plurality of sensors is operated in rolling shutter modeto read out an as yet unread row of pixel values corresponding to thecurrent scan position when the X^(th) sensor has an unread row of pixelvalues corresponding to the current scan position. Accordingly, in thismanner the entire scene of interest and/or various portions of the sceneof interest are scanned by the various sensors of the camera device.

In some embodiments the first and second sensors correspond to first andsecond portions of the scene area of interest, respectively, the extentof the first portion of the scene area of interest being greater thanthe extent of the second portion of the scene area of interest in thedirection of scan. In some embodiments the direction of scan proceedsfrom top to bottom. In some embodiments the first sensor corresponds toa first optical chain having a first focal length and the second sensorcorresponds to a second optical chain having a second focal length. Insome embodiments the controlling step 4104 further includes performingstep 4112 to control the duration of read out of the one or more imagesensors. In step 4112 the duration of readout of the first sensor iscontrolled as a function of first sensor image capture size and/or focallength of a first optical chain to which the first sensor corresponds.In some embodiments, step 4112 includes performing one of the steps 4114or 4116 as part of controlling the duration of sensor readouts. In step4114 the duration of readout of the first sensor is controlled to have alonger duration than the readout of the second sensor when the focallength of the first optical chain is smaller than the focal length ofthe second optical chain. In some embodiments, the duration of thereadout of the first sensor is controlled to have a longer duration thanthe readout of the second sensor when the first and second sensorscorrespond to first and second portions of the scene area of interest,respectively, and the extent of the first portion of the scene area ofinterest in the direction of scan is greater than the extent of thesecond portion of the scan area of interest in the direction of thescan. This occurs for example, when the focal length of the firstoptical chain including the first sensor is smaller than the focallength of the second optical chain including the second sensor.

In some other embodiments the focal length of the first optical chain(to which the first sensor corresponds) is N times the focal length ofthe second optical chain. In some such embodiments step 4116 isperformed as part of step 4112. In step 4116 the duration of reading outof pixel values from the first sensor is controlled to include readingrows of pixel values out of the first sensor at a rate which is N timesfaster than the rate at which rows of pixel values are read out from thesecond sensor. In various embodiments a rolling shutter controller inthe camera device controls the scanning and read out operation discussedabove. In some embodiments the pixel size of the first sensor is thesame as the pixel size of the second sensor. In some embodiments thefirst and second sensors include the same total number of row andcolumns of pixels.

Operation proceeds from step 4101 (that includes one or more stepsdiscussed above) to step 4120 via connecting node A 4118. In step 4120the one or more images read out of the first through X^(th) sensors arestored, e.g., X images are stored, one corresponding to each of the Xsensors. Operation proceeds from step 4120 to step 4122. In step 4122,at least two or more or all of the captured images (read out from thesensor) are combined to generate a composite image in accordance withthe invention.

FIG. 42 shows the steps of a method 4200 of capturing a scene area ofinterest using a plurality of optical chains, e.g., camera modules, thatis implemented in some embodiments. The method starts in step 4202,e.g., with a user initiating the capture of a scene area of interest ona camera device which causes the camera device, e.g., camera device 100,to initiate a scan and thus image capture of the scene area of interestby one or more camera optical chains which are operated in a coordinatedmanner.

Operation proceeds to step 4204 in which the camera device initializes acurrent scan position, e.g., by setting the current scan position to ascan position starting value, e.g., 1, in the FIG. 42 example.

Once the scan position value is initialized, operation proceeds fromstep 4204 to step 4206, wherein a controller, e.g., rolling shuttercontroller 150, controls each of the image sensors of the camera deviceto perform a read out of pixel values, e.g., rows of pixel values, in asynchronized manner, e.g., with rows of pixel values being read outsequentially in accordance with operation of a rolling shutterimplementation.

Step 4206 includes image sensor read out control steps for each of theimage sensors 1 to X which are being used to capture at least a portionof a scene area of interest. For purposes of discussion steps 4210 and4214 are shown with the same or similar steps being performed for eachof the other image sensors being used to capture at least a portion of ascene area of interest.

In step 4210 a determination is made as to whether or not image sensor 1has an as yet unread row of pixels corresponding to the current scanposition, e.g., a row of pixel values which are to be read out. If imagesensor 1 does not have an unread row of pixel values corresponding tothe current scan position, e.g., because the image sensor corresponds toan image portion outside the area corresponding to the current scanposition, operation proceeds to step 4220 without a row of pixels beingread out from image sensor 1.

In step 4214 a determination is made as to whether or not image sensor Xhas an as yet unread row of pixels corresponding to the current scanposition, e.g., a row of pixel values which are to be read out. If imagesensor X does not have an unread row of pixel values corresponding tothe current scan position, e.g., because image sensor X corresponds toan image portion outside the area corresponding to the current scanposition, operation proceeds to step 4220 without a row of pixels beingread out from image sensor X.

It should be appreciated that by the time operation proceeds to step4220 from step 4206, each of the image sensors 1 through X which had anunread row of pixel elements corresponding to the current scan positionwill have read out the pixel values of the row corresponding to thecurrent scan position.

In step 4220 the current scan position is updated to the next scanposition, e.g., the current scan position is incremented by one. As thecurrent scan position is incremented, the scanning of the image moves tothe next scan line resulting in the scan lines passed through insequence, e.g., from the top of the image to the bottom of the imageassuming a top to bottom scan of a scene area.

Operation proceeds from step 4220 to step 4222 in which a determinationis made as to whether the current scan position that was just updated instep 4220 exceeds the last scan position value of the scene of interest.If the current scan position exceeds the last scan position of the scenearea of interest it indicates that the scene of interest has been fullyscanned and captured as a result of the image sensor readouts. However,if in step 4222 it is determined that the current scan position valuedoes not exceed the value last scan position value of the scene area ofinterest, operation proceeds to step 4206 so that unread pixel elementrows of image sensors having a row of pixel elements corresponding tothe current scene area position will be read out.

It should be appreciated that the focal length and area of a scene towhich a camera optical change and its corresponding image sensor aredirected will affect whether a row of pixel values of an image sensorare read out at a particular point in time in the FIG. 42 embodiment.Since image sensor read out is controlled based on scan position, thesensor readouts of multiple image sensors occurs in a coordinatedmanner. While the image sensor readout of pixel values is coordinated inthe FIG. 42 embodiment, the readout of some image sensors may occur,depending on the camera device configuration, before the readout of oneor more other sensors is completed.

Generally, in the FIG. 42 embodiment, the readout of an image sensorcorresponding to a camera optical chain having a smaller focal lengthand thus corresponding to a larger portion of the scene area of interestwill occur at a slower rate than the readout of an image sensor havingthe same number of rows of pixel elements but having a larger focallength and thus corresponding to a smaller portion of the overall scenearea of interest.

Once the full scene area of interest has been scanned, operationproceeds from step 4222 to step 4224 in which images, e.g., sets ofpixel values, read out of the first through X^(th) image sensors arestored in memory. Then in step 4224, the captured images are processed,e.g., by processor 110, to generate a composite image which may bestored and/or output, e.g., transmitted and/or displayed. The generationof the composite image performed in step 4226 may be performed in any ofa variety of ways including combining pixel values of images captured bydifferent image sensors using a weighted sum approach such as describedabove with respect to at least some image combining processes that can,and in various embodiments are, used to generate a composite image.

It should be appreciated that the method of FIG. 42 has been explainedin terms of reading out rows of pixel elements of an image sensor. Someimage sensors allow, depending on the mode of operation, pixel elementsof multiple pixel rows to be treated as a single row. Such a “joint”read results in a single pixel value which is the sum of the pixelvalues sensed by each of the jointly read rows being read out as asingle pixel value. In cases of such a “joint” read, the multiple rowsof pixel elements which are jointly read out as a single row of pixelvalues is treated as one row of pixel elements for purposes ofimplementing the method of FIG. 42. The joint read approach whileresulting in a reduced number of pixel values being read from an imagesensor may be desirable in embodiments where motion is an issue and itis desirable to capture an image in as little time as possible.

Generally, if different image sensors correspond to different focallengths, assuming each of the image sensors include the same number ofrows of pixel values the read out time to read out the full set of rowsof an image sensor in the FIG. 42 embodiment can be expressed as afunction of the focal length of the optical chain, e.g., camera module,in which the image sensor is included and the number of rows of pixelelements of the image sensor which will be jointly read out and treatedas a single row of pixel elements at a given time.

In some exemplary embodiment an imaging device such as e.g., the cameradevice 100, is used to implement the method of flowcharts 4100 and 4200.In one such embodiment the plurality of optical chains 130 of the cameradevice 100 include optical chains arranged in the manner as illustratedin FIG. 12A with more detailed arrangements and elements (e.g., sensors,filters) of the optical chains further shown in FIG. 12B. Thus in suchan embodiment the plurality of optical chains 130 include optical chains1202 through 1234 discussed with regard to FIG. 12A. In anotherembodiment the plurality of optical chains 130 of the camera device 100include optical chains of the type and arrangement as illustrated inFIG. 17A. In such an embodiment the plurality of optical chains 130include optical chains 1702 through 1734, some of which includenon-circular lenses, discussed with regard to FIGS. 17A and 17B.

In some embodiments the controller 150 (including the sensor read outcontroller 289) is configured to control a plurality of image sensors,corresponding to the plurality of optical chains 130, to scan a scenearea of interest from a first edge of the scene area of interest to asecond edge of the scene area of interest, the first and second edges ofthe scene area of interest being parallel, the sensor controller 289controlling, as part of the scan, image capture according to a scanposition which progresses from the first edge to the second edge of thescene area of interest. The sensor controller controls a first sensor inthe plurality of image sensors in rolling shutter mode to read out a rowof pixel values corresponding to a current scan position when the firstimage sensor has a row of pixel values corresponding to the current scanposition, the current scan position changing over time. In someembodiments the sensor controller 289 further controls a second imagesensor in the plurality of image sensors in rolling shutter mode to readout a row of pixel values corresponding to the current scan positionwhen the second image sensor has a row of pixel values corresponding tothe current scan position, the first and second image sensors capturingimages of different portions of the scene area of interest.

In some embodiments the first image sensor captures an image of aportion of the scene area of interest which is not captured by thesecond image sensor and the second image sensor captures an image of aportion of said scene area of interest which is not captured by saidfirst image sensor.

In some embodiments the plurality of image sensors further includes athird image sensor (e.g., corresponding to a third optical chain). Insome such embodiments the first image sensor captures a first portion ofsaid scene area of interest, the second image sensor captures a secondportion of the scene area of interest, the first and second portions ofthe scene area of interest being partially non-overlapping and the thirdimage sensor captures the entire scene area of interest. This can beappreciated by briefly referring to FIGS. 12 and 32. For example a firstportion 3206 is captured by a first image sensor corresponding to afirst optical chain, e.g., OC 1204, a second portion 3208 is captured bya second image sensor corresponding to a second optical chain, e.g., OC1208, and a third portion 3202 is captured by a third image sensorcorresponding to a third optical chain, e.g., OC 1228. It can beappreciated from the example of FIG. 32 that the first and secondportions (3206 and 3208) of the scene area of interest are partiallynon-overlapping and the third image sensor captures the entire scenearea of interest.

In one embodiment the first and second image sensors correspond to firstand second portions of the scene area of interest, respectively, theextent of the first portion of the scene area of interest being greaterthan the extent of the second portion of the scene area of interest inthe direction of scan, e.g., from top to bottom. In one such embodimentthe sensor controller 289 is configured to control the duration ofreadout of the first image sensor to have a longer duration than thereadout of the second image sensor.

In some embodiments the first image sensor corresponds to an opticalchain, e.g., OC 1204, having a first focal length and said second imagesensor corresponds to an optical chain, e.g., OC 1234, having a secondfocal length. In some such embodiments the controller 289 is configuredto control the rate at which pixel rows are read out of the first andsecond image sensors as a function of the focal length of the individualoptical chains to which the first and second image sensors correspond.In some embodiments the pixel size of the first image sensor is the sameas the pixel size of the second image sensor. In some embodiments thefirst focal length is N times the second focal length. In suchembodiments the sensor controller is further configured to control theduration of reading out of pixel values from the first image sensor at arate which is N times faster than the rate at which rows of pixel valuesare read out from the second image sensor. In some embodiments the firstand second image sensors include the same total number of rows andcolumns of pixels.

In some embodiments the plurality of image sensors (corresponding to theplurality of optical chains 130) include multiple image sensors whichcapture substantially non-overlapping portions of the scene area ofinterest and a union of the portions of the scene area of interestcaptured by the multiple sensors cover the scene area of interest. Insome embodiments multiple image sensors include an even number ofsensors. In some embodiments, the even number of image sensors is four.

In one embodiment a first portion of the scene area of interest capturedby the first image sensor begins at a scan location which precedes asecond portion of the scene area of interest captured by the secondimage sensor. In one embodiment the second portion of the scene area ofinterest corresponds to the center of said scene area of interest. Insome such embodiments the plurality of image sensors (corresponding tothe plurality of optical chains) includes 5 image sensors, 4 of whichcapture substantially non-overlapping portions of the scene area ofinterest. In some such embodiments first through fourth different cornerportions of the second portion of the scene area of interest overlapdifferent corresponding portions of the scene area of interest which arecaptured by the remaining four sensors of said five sensors. Thisarrangement can be appreciated by briefly referring to FIG. 32. It canbe appreciated that four corners of the scene area 3216 (in the center)overlaps the different portions (scene areas 3206, 3208, 3212, 3214) ofa scene area of interest captured by four sensors (corresponding to fouroptical chains).

In some embodiments, the plurality of sensors includes 5 sensors, 4 ofwhich capture four different portions of the scene area of interest, atleast one corner of each of the four different portions of the scenearea of interest not overlapping an image portion captured by the otherthree of the four different portions of the scene area of interest, thefifth sensor capturing a center portion of the scene area of interest.

In various embodiments the processor 110 initializes a current scanposition, e.g., by setting the current scan position to a scan positionstarting value, prior to starting of a sensor read out operation asdiscussed with regard to FIG. 42. In various embodiments the exposureand read out controller 150 (that includes sensor read out controller289) alone, or under direction of the processor 110, controls each ofthe image sensors of the camera device to perform a read out of pixelvalues, e.g., rows of pixel values, in a synchronized manner. In someembodiments the controller 150 is further configured to determine as towhether or not one or more image sensors have an unread row of pixelscorresponding to the current scan position, e.g., a row of pixel valueswhich are to be read out. When it is determined that not one or moreimage sensors have an unread row of pixels corresponding to the currentscan position, the controller 150 controls each of the one or more imagesensors to perform a read out of the row of pixel values. In someembodiments the processor 110 is further configured to update thecurrent scan position to the next scan position, e.g., incrementing thecurrent scan position by one.

It should be appreciated that various features and/or steps of method4100 and 4200 relate to improvements in cameras and/or image processingeven though such devices may use general purpose processors and/or imagesensors. While one or more steps of the method 4100 and 4200 have beendiscussed as being performed by a processor, e.g., processor 110, 211,it should be appreciated that one or more of the steps of the method4100 and 4200 may be, and in some embodiments are, implemented bydedicated circuitry, e.g., ASICs, FPGAs and/or other applicationspecific circuits which improve the efficiency, accuracy and/oroperational capability of the imaging device performing the method. Insome embodiments, dedicated hardware, e.g., circuitry, and/or thecombination of dedicated hardware and software are utilized inimplementing one or more steps of the method 4100 and 4200 thereinproviding additional image processing efficiency, accuracy and/oroperational capability to the imaging device, e.g., camera, implementingthe method. FIG. 43 shows the steps of a method 4300 of capturing ascene area of interest using a plurality of optical chains, e.g., cameramodules, in a synchronized manner in accordance with another exemplaryembodiment. For purposes of discussion, the capturing of imagescorresponding to different scene areas performed by image sensors ofdifferent optical chains will be explained by referring to the opticalchains illustrated in FIG. 12A and using that as the basis of theexample. As previously discussed in FIG. 12A example, three differentfocal lengths, f1, f2 and f3 are used where f1<f2<f3, the same focallength relationship is considered for discussion.

The method 4300 starts with step 4302, e.g., when a user of a cameradevice, e.g., camera 100, presses a button or takes another action totrigger the capture of an image of a scene area of interest. In step4304 the exposure and read out controller 150 (that includes sensor readout controller 289) alone, or under direction of the processor 110,controls a plurality of optical chains 130 to read out pixel valuescorresponding to a portion of the scene area of interest at the sametime, e.g., concurrently. It should be appreciated that while aplurality, e.g., two or more, of the optical chains 130 are operated toread out at the same time, one or more of the remaining optical chains130, e.g., optical chains which are directed to a different portion ofthe scene of interest, may not be controlled to read out the pixelvalues while other image sensors are reading out pixel values. In atleast one implementation of the method 4300, the plurality of opticalchains 130 includes a first optical chain, e.g., optical chain 1204)having a first focal length (f2) and a second optical chain, e.g.,optical chain 1234, having a second focal length (f1). As part of thereading out of the image sensors, different portions of the scene areaof interest will be read out at different times, e.g., as the rollingshutter and read out move down from the top of the image sensor to thebottom of the image sensor where the top of the image sensor correspondsto the start of the rolling shutter readout and the bottom of the imagesensor corresponds to the last row to be read out. To facilitatesynchronization of sensor readouts, in at least one embodiment the topsand bottoms of image sensors of the different optical chains arearranged in the same direction so that the read out of rows of pixelvalues can be easily synchronized. However, the use of a uniform scanand thus readout direction for the different image sensors of theoptical chains is not mandatory for all embodiments even though it isused on at least some embodiments.

Since the image sensors are controlled to perform pixel row readoutssequentially, e.g. in accordance with the implemented rolling shuttercontrol, different portions of the scene area of interest will be readout from sensors of one or more optical chains at different times whilethe readout of pixel values corresponding to a portion of the scene areaof interest will occur in a synchronized manner with the sensor ofmultiple optical chains reading out rows of pixel values correspondingto the same portion of a scene area of interest in parallel.

Step 4304 includes various sub steps in at least some embodiments. Step4304 includes step 4306 of implementing a synchronized rolling shutterread out of the first image sensor included in the first optical chainand the second image sensor included in the second optical chain. Thus,the first and second image sensors of the first and second opticalchains will be read out in a synchronized fashion taking intoconsideration the operation of the timing of a first rolling shutterused to control the read out of the first image sensor of the firstoptical chain and the timing of a second rolling shutter used to controlthe read out of the second image sensor, i.e., the image sensor of thesecond optical chain.

As part of step 4306, steps 4308 and 4310 maybe, and in variousembodiments are, performed. In step 4308 the first image sensor of thefirst optical chains is controlled to sequentially read out rows ofpixel values beginning at the top edge of the first image sensor andproceeding to a bottom edge of the first image sensor where the top edgeof the first image sensor of the first optical chain corresponds to arow of pixel values that are read out first as part of the rollingshutter controlled read out of the first image sensor. In step 4310 thesecond image sensor of the second optical chain is controlled tosequentially read out rows of pixel values beginning at the top edge ofthe second image sensor. The readout of the second image sensor proceedsto a bottom edge of the second image sensor where the top edge of thesecond image sensor of the second optical chain corresponds to a row ofpixel values that are read out first as part of the rolling shuttercontrolled read out of the second image sensor. In this example, thefirst and second image sensors are configured so that they both havetheir tops and bottoms oriented in the same direction relative to thetop of the camera device, e.g., camera device 100, implementing themethod.

While the method of 4300 is explained with respect to first and secondimage sensors of first and second optical chains, it should beappreciated that in some embodiments the method is implemented for morethan two optical chains, e.g., for Z optical chains where Z can beanywhere in the range of from 2 to Z, Z being an integer. In such inimplementation, step 4306 includes a read out step similar to steps4308, 4310 for each of the individual Z optical chains and correspondingimage sensors of the camera device in which the method is implemented.

In addition to step 4306, step 4304 includes step 4312. In step 4312,the read out rate of the first and second image sensors is controlled asa function of the focal length of the first and second optical chains,respectively. In some embodiments where the focal length of the firstoptical chain (f2) is larger than the focal length of the second opticalchain (f1), the image sensor readout controller 289 controls the firstimage sensor to read out a number of rows of pixels in the first imagesensor in a period of time which is calculated as the ratio of thesecond focal length (f1) to the first focal length (f2) times the amountof time used to read out the same number of rows of pixels in saidsecond image sensor.

Consider for example the case where the first and second image sensorshave the same number of rows of pixel elements but the first imagesensor corresponds to the first optical chain 1204 having a first focallength (f2) and a second optical chain, e.g., module 1234, having asecond focal length (f1) where f2 is twice f1. As previously discussed,in such a case the second optical chain (f1) will capture a portion ofthe scene area which is four times or approximately four times the sizeof the scene area captured by the first optical chain (f2). Tosynchronize the image capture process assuming that the image sensor ofthe first optical chain (f2) is controlled to read its full set of pixelvalues in time NT_(R), (where N is the number of pixel element rows onthe first and second image sensors and T_(R) is the time required toread a single row of pixel elements on either of the image sensors), thesecond image sensor with the shorter focal length (f1) will becontrolled to read out its rows of pixel values over the time period2NT_(R) assuming all of the rows of the image sensor are to be read outindividually.

Thus assuming image sensors have the same number of rows, the time overwhich an image sensor will be controlled to read out its rows of pixelvalues is as follows:

Sensor 1 with N rows of pixel elements and a focal length of f2 willhave a read out period of (f2/f2)NT_(R)

Sensor 2 also with N rows pixel elements and a focal length of f1(smaller than sensor 1 focal length f2) will have a read out period of:NT_(R) (f2/f1)=2NT_(R) (since f2=2f1), assuming each of the individualrows of pixel values are individually read out.

Note that f2 is used in this example as the numerator since it is thelargest focal length being used in the example.

As noted above, in some modes of operation a joint read out of rows ofpixel elements may be implemented to reduce amount of time required toread out an image sensor. While in various embodiments where eachsensor's rows of pixel elements are read out individually to maximizethe overall pixel count of a generated composite image, and the read outrate of the sensor corresponding to the smaller focal length is reducedto maintain synchronization with the read out of pixel values and thusthe capture of image portions of the scene of interest of the sensorcorresponding to the optical chain having the larger focal length thisdoes not occur in all embodiments.

For example in one particular exemplary embodiment where capture of fastmotion is desired, rather than slowing down the row read out rate of theimage sensor corresponding to the shorter focal length, a joint read outoperation is performed for rows of the image sensor corresponding to thelarger focal length. As noted above, in a joint readout multiple rows ofpixel values are read out jointly with, for example two rows of pixelvalues providing the same number of pixel values as a single row when ajoint read of the two rows is implemented. While a joint read by afactor of M will result in a reduction of rows of pixel values by thesame factor, the image sensor read out will be completed in 1/Mth thetime.

In at least one embodiment, rather than slow down the read out rate ofone or more sensors corresponding to optical chains having a short focallength, e.g., a focal length shorter than the longest focal length to beused to capture an image that will be combined to form a compositeimage, a joint read out of pixel rows of the sensor with the longerfocal length is implemented with the number of rows being jointly readout being controlled as a function of the difference in the focallengths. For example if f2 is twice f1, in one embodiment M is set equalto 2 so that the read out of the sensor corresponding to the f2 focallength can be completed in half the time used to read out the rows ofpixel values from the second sensor corresponding to the optical chainwith shorter focal length f1.

In such a case the time required to read out the pixel rows of thesensor with the shortest focal length serves as the time constraint forthe multiple image sensors to complete their read out and sensors withlarger focal lengths have the number of rows read out reduced by use ofa joint read operation based on a factor which is determined by theratio of the larger focal length to the smaller focal length. Forexample if f2 is larger than f1 by a factor of 2, sensors correspondingto optical chains with an f2 focal length will be read out using jointreads implemented by a joint read factor of 2.

In this way, the image sensors of the optical chains with differentfocal length can be controlled to capture the same portions of a scenearea of interest with the total capture time being limited to the timerequired to read a single image sensor.

While the joint read reduces the number of pixel values which will beread out of the image sensor with the larger focal length, the timerequired to perform the read is reduced and the total pixel count of theimage will not be reduced below that of the optical chain with theshorter focal length f1.

In at least one embodiment a user is provided the opportunity to selectbetween a motion mode of operation in which joint reads are used toreduce or minimize the image capture time period, e.g., to keep thetotal time used to capture an image to NT_(R) which corresponds to thetime required to read a single image sensor in some embodiments, and astill or slow motion mode of operation where the pixel count of an imageis optimized at the expense of the image capture time, e.g., anembodiment where the read out time period of pixel element rowscorresponding to the image sensor of the optical chain with the shorterfocal length is increased to achieve timing synchronization with scenearea capture by optical chains having a larger focal length.

Taking into consideration the potential use of the joint row read factorM, the total time used to read out rows of pixel values from first andsecond image sensors having the same number of rows of pixel elementsbut corresponding to different focal length f1 and f2, respectivelywhere f2 is greater than f1, can be expressed as follows:

-   -   Sensor 2 with N rows of pixel elements and corresponding to        optical chain having a focal length of f1 (shorter focal length)        will have a read out period of NT_(R);    -   Sensor 1 also with N rows pixel elements but corresponding to        the optical chain having a focal length of f2 (larger) is        controlled to perform a joint read of rows by a factor of M will        have a read out period of: NT_(R)/M, where M is equal to (f2/f1)        assuming one row of pixel values is read out for each M rows of        pixel elements.

While sensor 1 (larger focal length f2) will complete a readout in lesstime than sensor 2, it should be appreciated that in various embodimentsanother sensor S3 with the same focal length (f2) as the optical chainto which sensor 1 corresponds, but which captures an image area, interms of scan direction beneath the area captured by sensor 1, willcapture the lower portion of the image. Thus, assuming f2 is twice f1,the read out of sensors S1 and S3 will coincide to the same time periodNT_(R) in which sensor 2 is read out but with the sensor 1 S1 read outoccurring during the first half of NT_(R) and the sensor 3 S3 readoutoccurring in the second half of NT_(R).

From the above discussion, it should be appreciated that step 4312 whichinvolves controlling the read out rate of the first and second imagesensors as a function of the focal length of the first and secondoptical chains may also depend on the mode of operation selected by theoperator of the camera, e.g., still or high speed motion. The selectionof which mode may, and in some embodiments is, made automatically basedon detected motion in one or more sequentially captured images orportions of images captured at different points in time. For example, ifa motion of one or more objects is detected which is likely to causeartifacts due to the motion which are likely to reduce the image qualityof a combined image more than reducing the overall pixel count of theimage would, the joint read approach to reading out pixel rows is used.

In the case of video, the camera device may, and in some embodimentsdoes, make updated decisions as to which mode of operation to operate into produce composite images of the best quality. As the result of saiddecisions, the camera device may switch between modes of operation andthe image capture synchronization technique used from one frame time tothe next frame time. While this may result in different composite imagescorresponding to the same video sequence having different numbers ofpixels, overall image quality is maximized.

While such dynamic switching is supported in some embodiments, in otherembodiments once a synchronization technique and mode of operation isdetermined, the mode is maintained for the video sequence so that thenumber of pixel elements of the images in the video sequence will remainconstant on a per composite image basis.

As should be appreciated in the case of synchronized readout of multipleimage sensors where a third sensor captures an image portion of a scenearea of interest which is located below (in terms of scan direction) theimage portion captured by another image sensor, e.g., first imagesensor, the readout of pixel values from the third image sensor willfollow the read out of pixel values from the first image sensor.

Thus, in at least one embodiment, step 4304 includes step 4314 wherein athird image sensor of a third optical chain, e.g., optical chain 1218,is controlled to read out pixel values corresponding to a third portionof the scene of interest after a last row of pixel values is read outfrom the first image sensor. In some embodiments the third portion ofsaid scene area of interest is positioned below (in the direction ofscan) said first portion of said scene area of interest. In some suchembodiments the first image sensor captures a first portion of the scenearea of interest and the second image sensor captures a second portionof the scene area of interest which is larger than the first portion. Insome such embodiments the second image sensor (corresponding to smallerfocal length optical chain) captures substantially the same image areaas the first and third image sensors combined but during a readout timewhich is longer than the read out time of either of the first and thirdimage sensors.

It should be appreciated that in such an embodiment while pixel valuescorresponding to the first image sensor may be read out during the timeperiod during which pixel values are read out from the second imagesensor, read out times of the first and third image sensors will notoverlap.

However, in embodiments where the first and third image sensors arecontrolled to capture overlapping portions of the scene of interest, theread out of pixel rows corresponding to an overlapping scene portion maybe read out at the same time from the first and third image sensors.

From step 4304 operation proceeds to step 4316 where the images read outfrom the image sensors which were controlled in step 4304 are stored inmemory for subsequent processing. Operation then proceeds to step 4318.In step 4318 the images captured by the different optical chains in asynchronized manner, e.g., in step 4304, are combined to generate acomposite image.

In the case of video, steps 4304, 4316 and 4318 will be performed foreach frame time, e.g., with one composite image being generated for eachframe time.

In some embodiments the operation proceeds from step 4318 to step 4320.In step 4320 the composite image generated in step 4318 is stored inmemory, transmitted and/or displayed, e.g., on the display 102.

While in some embodiments the generation of the composite image occursin the camera device 100 with the composite image then being displayedor transmitted, in other embodiments the composite image is generated bya processor, e.g., as part of a post capture processing process and/orvideo production processes. In such embodiments the generation of thecomposite image may be done on a computer system including memory, aprocessor and a display which is different from the memory, processorand display of the camera device 100 including the optical chains 130used to capture the images of the scene area of interest.

In some exemplary embodiments an imaging device such as e.g., the cameradevice 100, is used to implement the method of flowchart 4300. In onesuch embodiment the plurality of optical chains 130 of the camera device100 include optical chains arranged in the manner as illustrated in FIG.12A with more detailed arrangements and elements of the optical chainsfurther shown in FIG. 12B. In another embodiment the plurality ofoptical chains 130 of the camera device 100 include optical chains ofthe type and arrangement as illustrated in FIG. 17A.

In one embodiment the controller 150 (including the image sensor readout controller 289) is configured to control the plurality of opticalchains to read out pixel values corresponding to a portion of said scenearea of interest at the same time, the plurality of optical chains 130including a first optical chain 1204 having a first focal length (f2)and a second optical chain 1234 having a second focal length (f1), thesecond focal length being different from the first focal length,different portions of said scene area of interest being read out atdifferent times. In various embodiments the controller 150 is configuredto implement synchronized rolling shutter read out of a first imagesensor included in the first optical chain 1204 and a second imagesensor included in the second optical chain 1234.

In some embodiments the controller 150 is configured to first read a topedge of the first image sensor of the first optical chain correspondingto a row of pixel values as part of implementing the rolling shutterread out of the first image sensor, the rolling shutter read outincluding sequential reading out of rows of pixel values beginning atthe top edge of the first image sensor and proceeding to a bottom edgeof the first image sensor. In various embodiments the controller 150 isfurther configured to first read a top edge of the second image sensorof the second optical chain corresponding to a row of pixel values aspart of a rolling shutter read out of the second image sensor, therolling shutter read out of the second image sensor including sequentialreading out of rows of pixel values beginning at the top edge of thesecond image sensor and proceeding to a bottom edge of the second imagesensor, the scan direction of the first and second image sensors (fromtop to bottom) being the same. While a top to bottom direction of scanof the image sensors is considered in the example discussed with regardto FIG. 43, however it should be appreciated that scanning from one edgeof the image sensor to another in a different direction is possible aspart of the rolling shutter read out of the image sensors.

In some embodiments, the first and second image sensors include the samenumber of rows of pixels. In some embodiments the controller 150 isconfigured to control the read out rate of the first and second imagesensors as a function of the focal length of the first and secondoptical chains, as part of being configured to control a plurality ofoptical chains to read out pixel values corresponding to a portion ofthe scene area of interest at the same time.

In some embodiments the first focal length is larger than the secondfocal length. In some embodiments the controller 150 is furtherconfigured to control the first image sensor to read out a number ofrows of pixels in the first image sensor in a period of time which iscalculated as the ratio of the second focal length to the first focallength times the amount of time used to read out the same number of rowsof pixels in the second image sensor. For example consider that thefocal length of the second optical chain is FS2 and the focal length ofthe first optical chain is FS1, both first and second sensorscorresponding to the first and second optical chains respectively,having N rows of pixel elements. If it takes time TS2 to read N rows outof sensor 2 then the controller 150 controls the read out of sensor 1 ina time period TS1=(FS2/FS1)×TS2.

In some embodiments the first focal length is twice the second focallength. In such embodiments the controller 150 is further configured tocontrol the second image sensor to be fully read out over a time periodwhich is twice as long as a first time period used to read out the firstimage sensor.

In various embodiments the first image sensor captures a first portionof the scene area of interest and the second image sensor captures asecond portion of the scene area of interest which is larger than thefirst portion. In some such embodiments the controller 150 is furtherconfigured to control a third image sensor of a third optical chain,e.g., optical chain 1218, to read out pixel values corresponding to athird portion of the scene area of interest after a last row of pixelvalues is read out from the first image sensor, the third portion of thescene area of interest being positioned below the first portion of thescene area of interest. In some embodiments the second image sensorcaptures at least substantially the same image area as the first andthird image sensors capture but during a readout time which is longerthan an individual read out time of either of the first and third imagesensors.

In various embodiments the images captured by the sensors as part of thesensor read out operations are stored in the device memory, e.g., memory108. In some embodiments, one or more of these images are furtherprocessed as part of generating a composite image. In some embodimentsthe processor 110 is configured to generate a composite image bycombining two or more captured images. In some embodiments the processor110 is further configured to control storage of the generated compositeimage in the memory 108 and/or output of the composite image on thedisplay 102 and/or transmission of the captured images or the compositeimage to another device via an interface such as interface 114.

It should be appreciated that various features and/or steps of method4300 relate to improvements in cameras and/or image processing eventhough such devices may use general purpose processors and/or imagesensors. While one or more steps of the method 4300, e.g., such ascomposite image generation step, have been discussed as being performedby a processor, e.g., processor 110, 211, it should be appreciated thatone or more of the steps of the method 4300 may be, and in someembodiments are, implemented by dedicated circuitry, e.g., ASICs, FPGAsand/or other application specific circuits which improve the efficiency,accuracy and/or operational capability of the imaging device performingthe method. In some embodiments, dedicated hardware, e.g., circuitry,and/or the combination of dedicated hardware and software are utilizedin implementing one or more steps of the method 4300 therein providingadditional image processing efficiency, accuracy and/or operationalcapability to the imaging device, e.g., camera, implementing the method.

FIG. 44 shows a flowchart 4400 illustrating the steps of an exemplarymethod of capturing images using multiple optical chains in accordancewith one exemplary embodiment. The method includes controlling animaging device, e.g., such as that shown in FIG. 1, 6, 8 and/or 14, tocapture images in accordance with an exemplary embodiment. The cameradevice implementing the method of flowchart 4400 can and sometimes doesinclude the same or similar elements as the camera device of FIG. 1and/or 4A.

The method of flowchart 4400 can be, and in some embodiments is,performed using a camera device such as the camera 100 of FIG. 1. In onesuch embodiment the plurality of optical chains 130 of the camera device100 include optical chains, e.g., camera modules, arranged in the manneras illustrated in FIG. 12A with more detailed arrangements and elementsof the optical chains further shown in FIG. 12B. In another embodimentthe plurality of optical chains 130 of the camera device 100 includeoptical chains of the type and arrangement as illustrated in FIG. 17A.In such an embodiment the plurality of optical chains 130 includeoptical chains 1702 through 1734 discussed with regard to FIGS. 17A and17B.

The exemplary method starts in step 4402, e.g., with a user initiatingthe capture of a scene area of interest which causes the camera device,e.g., camera device 100, to initiate image capture of the scene area ofinterest by one or more optical chains. For the purposes of discussionconsider that the camera device includes a plurality of optical chains,and each of the optical chains can be independently operated andcontrolled.

Operation proceeds from step 4402 to steps 4404, 4406, 4408, 4410, 4412,4414 and 4416 which involve image capture operations. The image captureoperations may and in some embodiments are performed in a synchronizedmanner. In at least some synchronized embodiments the images captured bysome by not necessarily all of the different optical chains correspondto the same or an overlapping time period. In other embodiments imagecapture is not synchronized but multiple one of the captured images arecaptured during the same or an overlapping time period. In still otherembodiments as least some images are captured sequentially, e.g., inrapid succession. Sequential image capture may, and in some embodimentsare used for capturing images corresponding to different portions of ascene area.

In step 4404 a first optical chain of the camera device is used tocapture a first image of a first portion of a scene area of interest,the first optical chain having a first optical axis and a firstoutermost lens. In some embodiments the scene area of interest may beslightly smaller than the full image capture area. Operation proceedsfrom step 4404 to step 4418. Step 4418 is performed in some but notnecessarily all embodiments. In some embodiments where step 4418 isskipped operation proceeds directly to step 4420.

In step 4406 a second image of a second portion of the scene area iscaptured using a second optical chain of the camera device, the secondoptical chain having a second optical axis which is not parallel to thefirst optical axis and a second outermost lens which is separate fromthe first outermost lens. Thus it should be appreciated that in variousembodiments each optical chain has a physically disjoint andnon-overlapping entrance pupil due to each optical chain having adifferent outer lens. In some embodiments the first and second opticalaxis are not perpendicular to the front face of the camera. In someembodiments the first and second optical chains have the same focallength. Operation proceeds from step 4406 to step 4418.

In step 4408 a third image including the scene area of interest, e.g.,the entire scene area of interest, is captured using a third opticalchain of the camera device having a third outermost lens which isseparate from said first and second outermost lens. In some embodimentsthe third optical chain has a third optical axis which is not parallelto either of the first or the second optical axis. In some embodimentsthe focal length of the third optical chain is smaller than a focallength of at least one of the first and second optical chains. In someembodiments the focal length of the third optical chain is smaller thanthe focal length of the first optical chain. In some embodiments thefocal length of the third optical chain is smaller than the focal lengthof the second optical chain. In some embodiments the focal length of thethird optical chain is smaller than the focal length of the first andthe second optical chains. In some embodiments the third optical axis isperpendicular to the front face of the camera device. In someembodiments the first optical chain captures the first image using afirst sensor and the third optical chain captures the third image usinga third sensor. Operation proceeds from step 4408 to step 4418.

In step 4410 a fourth image is captured using a fourth optical chain ofthe camera device having a having a fourth outermost lens which isseparate from said first, second and third outermost lens, the fourthimage including a second image of the scene area of interest. In someembodiments the third optical chain has a third optical axis, and thefourth optical chain has a fourth optical axis. In some embodiments thethird and fourth optical axes are parallel to each other. In someembodiments the third and fourth optical axes are not parallel to thefirst or second optical axes. In some embodiments the third and fourthoptical axis are perpendicular to the front face of the camera. In someembodiments the fourth optical chain has a focal length which is thesame as the focal length of the third optical chain. Operation proceedsfrom step 4410 to step 4418.

In step 4412 a fifth image is captured using a fifth optical chain ofthe camera device having a fifth optical axis, the fifth image being animage of a third portion of the scene area of interest, the fifthoptical axis not being parallel to the first and second optical axes.Thus the fifth optical chain captures a part, e.g., quarter or half,portion of the scene area of interest. Operation proceeds from step 4412to step 4418.

In step 4414 a sixth image is captured using a sixth optical chain ofthe camera device having a sixth optical axis which is not parallel tothe first, second, or fifth optical axis, the sixth image being an imageof a fourth portion of the scene area of interest. Operation proceedsfrom step 4414 to step 4418. In some embodiments the first, second,fifth and sixth images each have a first number of pixel values.

In step 4416 a seventh image of a seventh scene area is captured using aseventh optical chain of the camera device having the same focal lengthas the first optical chain, the seventh optical chain having an opticalaxis perpendicular to the face of the camera. In some embodiments theseventh scene is at the center of the scene area of interest. Operationproceeds from step 4416 to step 4418.

Returning now to step 4418. In step 4418 one or more captured images,e.g., first, second, third, fourth, fifth, sixth and/or seventh images,are stored, e.g., in a device memory and/or output, e.g., to a displaydevice and/or to an external device via an interface. In someembodiments the processor 110 controls storage of the one or morecaptured images in the memory 108 and/or outputting of the one or morecaptured images to the display 102. While step 4418 is performed in someembodiments, in some other embodiments step 4418 may be skipped. In suchembodiments the operation proceeds from the previous step directly tostep 4420.

In various embodiments the second, third, fifth, and sixth opticalchains are arranged in such a manner that the images of the portion ofthe scene area of interest taken by these optical chains are fromdifferent spatially separated entrance pupils and thus have differentperspectives. Combining such images with different perspectivesintroduces artifacts, e.g., parallax. To minimize and/or alleviate theeffect of such artifacts from a composite image generated using thevarious images captured by these different optical chains, in someembodiments depth information is used which provides for parallaxcorrection when combing the images to avoid distortions of the compositeimage due to the different perspectives.

Returning to step 4420. In step 4420 a composite image is generated bycombining at least two of the captured images, e.g., the first andsecond images. In various embodiments the composite image is generatedby, e.g., a processor such as processor 110 or 211, of the camera deviceimplementing the steps of the method 4400. In some embodimentsperforming step 4420 includes performing one or more of steps 4424 and4426. In some embodiments sub-steps 4424 and 4426 are differentalternatives any one of which may be performed as part of implementingstep 4420. In some embodiments step 4420 includes performing step 4424where at least said first, second, third and fourth images are combinedas part of generating the composite image. In such embodiments theprocessor 110 is configured to generate the composite image by combiningat least the first, second, third and fourth images. In some embodimentsthe composite image is a composite image of the scene area of interest,the composite image having a larger number of pixel values than eitherof the first, second, third and fourth images individually have.

In some embodiments step 4420 includes performing step 4426 where thecomposite image is generated, e.g., by the processor 110, from at leastsaid first, second, third, fourth, fifth, and sixth images, thecomposite image having a number of pixel values more than three timesthe first number of pixel values but less than six times the firstnumber of pixel values. In some embodiments the composite image includesa number of pixel values which is less than the number of pixel valuesresulting from the sum of the number of pixel values combined togenerate the composite image. In various embodiments the generatedcomposite image is an image of the scene area of interest. In someembodiments the composite image is generated from the first, second,third, fourth, fifth, sixth images and seventh images.

In some embodiments the operations proceeds from step 4420 to step 4428.In step 4428 the generated composite image is stored, e.g., in a devicememory and/or output, e.g., to a display device and/or to an externaldevice via an interface. In some embodiments the storage of thecomposite image in the memory 108 and/or output, e.g., to a displaydevice and/or to an external device via an interface is performed underthe control of the processor 110.

It should be appreciated that various features and/or steps of method4400 relate to improvements in cameras and/or image processing eventhough such devices may use general purpose processors and/or imagesensors. While one or more steps of the method 4400, e.g., such ascomposite image generation step, have been discussed as being performedby a processor, e.g., processor 110, 211, it should be appreciated thatone or more of the steps of the method 4400 may be, and in someembodiments are, implemented by dedicated circuitry, e.g., ASICs, FPGAsand/or other application specific circuits which improve the efficiency,accuracy and/or operational capability of the imaging device performingthe method. In some embodiments, dedicated hardware, e.g., circuitry,and/or the combination of dedicated hardware and software are utilizedin implementing one or more steps of the method 4400 therein providingadditional image processing efficiency, accuracy and/or operationalcapability to the imaging device, e.g., camera, implementing the method.

FIG. 45 shows a flowchart 6700 illustrating the steps of an exemplarymethod of capturing images. The images may, and in some embodiments arecaptured using a plurality of optical chains included in a camera deviceincluding a camera housing with a front surface and a rear surfacehaving a thickness D, D being a distance between the front surface andthe rear surface. In one embodiment the method of flowchart 6700 can beimplemented to capture images using a camera device such as, e.g., thecamera device 1320 shown in FIG. 13B. The camera device includes aplurality of optical chains, and each of the optical chains can beindependently operated and controlled. The camera device implementingthe method of flowchart 6700 can and sometimes does include the same orsimilar elements as the camera device of FIGS. 1 and/or 4A. Thus itshould be appreciated that the camera device implementing the method6700 includes a processor, memory, interface(s) and other elementsdiscussed with regard to FIGS. 1 and 4A.

The exemplary method starts in step 6702, e.g., with a user initiatingthe capture of a scene area of interest which causes the camera device,e.g., camera device 1320, to initiate image capture of the scene area ofinterest by one or more optical chains. Operation proceeds from step6702 to steps 6704, 6706, 6708, and 6710 which involve image captureoperations. The image capture operations may and in some embodiments areperformed in a synchronized manner. In at least some synchronizedembodiments the images captured by some by not necessarily all of thedifferent optical chains correspond to the same or an overlapping timeperiod. In other embodiments image capture is not synchronized butmultiple one of the captured images are captured during the same or anoverlapping time period. In still other embodiments as least some imagesare captured sequentially, e.g., in rapid succession. Sequential imagecapture may, and in some embodiments are used for capturing imagescorresponding to different portions of a scene area.

In step 6704 a first image is captured, using a first optical chain inthe camera housing, the first optical chain including i) a first lightredirection device, ii) a first lens having a non-circular aperture, andiii) a sensor, the first optical chain having an optical axis includinga first optical axis portion in front of the light redirection deviceand a second optical axis portion extending from the light redirectiondevice to the sensor, the first lens being on the second optical axisportion, the non-circular aperture having a length less than or equal toD in a first direction along the direction of the thickness of thecamera and a length larger than D along a second direction perpendicularto the first direction. In some embodiments the first lens is the lensclosest to the light redirection device on the second optical axisportion. The various elements and/or constructional features of theexemplary camera device that can be used to implement the method offlowchart 6700 can be appreciated from FIG. 13B that shows such featuressuch as camera housing 1322, thickness 1323, front and rear surface 1321and 1325. Operation proceeds from steps 6704 to step 6712.

In step 6706 a second image is captured, using a second optical chain inthe camera housing, the second optical chain including i) a second lightredirection device, ii) a second lens having a second non-circularaperture, and iii) a second sensor, the second optical chain having anoptical axis including a third optical axis portion in front of thesecond light redirection device and a fourth optical axis portionextending from the second light redirection device to the second sensor,the second non-circular aperture having a length less than or equal to Din the first direction along the thickness of the camera and a lengthlarger than D along a third direction perpendicular to the direction ofthe thickness of the camera, the second and third directions beingdifferent. In some embodiments the second and third directions are at anangle of 90 degrees with respect to each other. Operation proceeds fromsteps 6706 to step 6712.

In step 6708 a third image is captured using a third optical chain, thethird optical chain including i) a third light redirection device, ii) athird lens having a third non-circular aperture, and iii) a thirdsensor, the third optical chain having an optical axis including a fifthoptical axis portion in front of the third light redirection device anda sixth optical axis portion extending from the third light redirectiondevice to the third sensor, the third non-circular aperture having alength less than or equal to D in the first direction along thethickness of the camera and a length larger than D along a fourthdirection perpendicular to the direction of the thickness of the camera,the first, second and third directions being different. In someembodiments the second and third directions are at an angle of 90degrees with respect to each other and the second and fourth directionarea at an angle with respect to each other between 30 degrees and 60degrees. Operation proceeds from steps 6708 to step 6712.

Similarly one or more additional optical chains in the camera device canbe used to capture additional images in some embodiments as indicated inthe figure. In some such embodiments where additional optical chains areused to capture additional images step 6710 is performed. In step 6710 aZ^(th) image is captured using a Z^(th) optical chain, the Z^(th)optical chain including i) a Z^(th) light redirection device, ii) aZ^(th) lens having a non-circular aperture, and iii) a Z^(th) sensor,the Z^(th) optical chain having an optical axis including an opticalaxis portion in front of the Z^(th) light redirection device and anotheroptical axis portion extending from the Z^(th) light redirection deviceto the sensor, the non-circular aperture having a length less than orequal to D in the first direction along the thickness of the camera anda length larger than D along a direction perpendicular to the directionof the thickness of the camera. Operation proceeds from steps 6710 tostep 6712. In some embodiments the various images captured by theplurality of optical chains as discussed above are captured in the sametime period.

Returning now to step 6712. In step 6712 one or more captured images,e.g., first, second, third etc., are stored, e.g., in a device memoryfor further processing in accordance with the features of the inventionand/or output, e.g., to a display device and/or to an external devicevia an interface. Operation proceeds from step 6712 to step 6714. Instep 6714 a composite image is generated by combining at least two ormore of the captured images. In various embodiments the composite imageis generated by, e.g., a processor such as processor 110 or 211, of thecamera device implementing the steps of the method 6700. In variousembodiments the generated composite image is an image of a scene area ofinterest. In some embodiments, at least some of the plurality of opticalchains capture portions of the scene area of interest, which may then becombined in accordance with the methods of the invention to generate acomposite image. Step 6714 is performed in some but not necessarily allembodiments.

Operation proceeds from steps 6714 to step 6716. In step 6716 thegenerated composite image is stored, e.g., in a device memory and/oroutput, e.g., to a display device and/or to an external device via aninterface.

It should be appreciated that various features and/or steps of method6700 relate to improvements in cameras and/or image processing eventhough such devices may use general purpose processors and/or imagesensors. While one or more steps of the method 6700, e.g., such ascomposite image generation step, have been discussed as being performedby a processor, e.g., processor 110, 211, it should be appreciated thatone or more of the steps of the method 6700 may be, and in someembodiments are, implemented by dedicated circuitry, e.g., ASICs, FPGAsand/or other application specific circuits which improve the efficiency,accuracy and/or operational capability of the imaging device performingthe method. In some embodiments, dedicated hardware, e.g., circuitry,and/or the combination of dedicated hardware and software are utilizedin implementing one or more steps of the method 6700 therein providingadditional image processing efficiency, accuracy and/or operationalcapability to the imaging device, e.g., camera, implementing the method.

FIG. 46 is a flowchart 6800 illustrating the steps of an exemplarymethod of capturing images. The images may, and in some embodiments are,captured using a camera device including a plurality of optical chains.In one embodiment the method of flowchart 6800 can be implemented tocapture images using a camera device such as, e.g., the camera device1400 shown in FIG. 14. The camera device includes a plurality of opticalchains, and each of the optical chains can be independently operated andcontrolled. The camera device implementing the method of flowchart 6800can and sometimes does include the same or similar elements as thecamera device of FIGS. 1 and/or 4A. The various elements and/or featuresof the exemplary camera device that can be used to implement the methodof flowchart 6800 can be appreciated from FIGS. 14-17 that shows suchfeatures such as the optical chains with non-circular aperture lenses,optical chains with round lenses, camera thickness etc.

The exemplary method starts in step 6802, e.g., with a user initiatingthe capture of a scene area of interest which causes the camera device,e.g., camera device 1400, to initiate image capture of the scene area ofinterest by one or more optical chains. Operation proceeds from step6802 to steps 6804, 6806, 6808, 6810, and 6812 which involve imagecapture operations. The image capture operations may and in someembodiments are performed in a synchronized manner. In at least somesynchronized embodiments the images captured by some by not necessarilyall of the different optical chains correspond to the same or anoverlapping time period. In other embodiments image capture is notsynchronized but multiple one of the captured images are captured duringthe same or an overlapping time period. In still other embodiments asleast some images are captured sequentially, e.g., in rapid succession.Sequential image capture may, and in some embodiments are used forcapturing images corresponding to different portions of a scene area.

In step 6804 a first image is captured, using a first optical chain ofthe camera, during a first time period, the first optical chain having afirst focal length and a first non-circular lens. Operation proceedsfrom steps 6804 to step 6814.

In step 6806 a second image is captured, using a second optical chain ofthe camera during the first time period, the second optical chain havinga second focal length and a second non-circular lens. In someembodiments the first and second focal lengths are the same. In someembodiments the first and second focal lengths are different. In someembodiments the first non-circular lens is longer in a first directionthan in a second direction which is perpendicular to the first directionand the second non-circular lens is longer in a third direction than ina fourth direction, the fourth direction being perpendicular to thethird direction, the first and third directions being different. In someembodiments the first optical chain includes a first sensor and thesecond optical chain includes a second sensor. Operation proceeds fromsteps 6806 to step 6814.

In step 6808 a third image is captured, using a third optical chainhaving a third focal length and including a round lens, the third focallength being less than the first or second focal lengths. In someembodiments the second and fourth directions are the same and correspondto the depth (thickness) of the camera. Operation proceeds from steps6808 to step 6814.

In step 6810 a fourth image is captured using a fourth optical chainhaving a fourth focal length and a third non-circular lens, the fourthfocal length being larger than the third focal length. In someembodiments the third non-circular lens is longer in a fifth directionthan in a sixth direction, the sixth direction being perpendicular tothe fifth direction, the first, third, and fifth directions beingdifferent by at least 20 degrees. Operation proceeds from steps 6810 tostep 6814.

Similarly one or more additional optical chains in the camera device canbe, and in some embodiments are, used to capture additional images. Inone embodiment additional images are captured using optical havingincluding circular lenses as indicated in step 6812. In step 6812additional images are captured, using a plurality of additional opticalchains including circular lenses, each of the additional optical chainsincluding circular lenses having a focal length less than the firstoptical chain. In some embodiments the various images captured by theplurality of optical chains as discussed above are captured in the sametime period. Operation proceeds from steps 6812 to step 6814.

Returning now to step 6814. In step 6814 one or more captured images,e.g., first, second, third etc., are stored, e.g., in a device memoryfor further processing in accordance with the features of the inventionand/or output, e.g., to a display device and/or to an external devicevia an interface. Operation proceeds from step 6814 to step 6816 whichis performed in some but not necessarily all embodiments. In step 6816 acomposite image is generated by combining at least two or more of thecaptured images, e.g., by combining at least the first and secondimages. In various embodiments the composite image is generated by,e.g., a processor such as processor 110 or 211, of the camera device1400 implementing the steps of the method 6800. In various embodimentsthe generated composite image is an image of a scene area of interest.In some embodiments, at least some of the plurality of optical chainscapture portions of the scene area of interest, which may then becombined in accordance with the methods of the invention to generate acomposite image.

Operation proceeds from steps 6816 to step 6818. In step 6818 thegenerated composite image is stored, e.g., in a device memory and/oroutput, e.g., to a display device and/or to an external device via aninterface.

It should be appreciated that various features and/or steps of method6800 relate to improvements in cameras and/or image processing eventhough such devices may use general purpose processors and/or imagesensors. While one or more steps of the method 6800, e.g., such ascomposite image generation step, have been discussed as being performedby a processor, e.g., processor 110, 211, it should be appreciated thatone or more of the steps of the method 6800 may be, and in someembodiments are, implemented by dedicated circuitry, e.g., ASICs, FPGAsand/or other application specific circuits which improve the efficiency,accuracy and/or operational capability of the imaging device performingthe method. In some embodiments, dedicated hardware, e.g., circuitry,and/or the combination of dedicated hardware and software are utilizedin implementing one or more steps of the method 6800 therein providingadditional image processing efficiency, accuracy and/or operationalcapability to the imaging device, e.g., camera, implementing the method.

FIG. 47 is a flowchart 4500 of an exemplary method of generating acomposite image in accordance with various exemplary embodiments. Theoperation of flowchart 4500 may be performed by a camera device or acamera device in combination with an image processing device. In someembodiments, the camera device is included is a camera. In someembodiments, the camera device is included in a wireless communicationsdevice, e.g., a smart phone. In some embodiments, the image processingdevice is a personal computer.

Operation of the exemplary method starts in step 4502, in whichinitialization is performed, and proceeds to step 4504. In someembodiments, operation also proceeds from step 4502 to one or more orall of optional steps 4516, 4518 and 4522.

In step 4504 a first optical chain of a camera device including a lightredirection device set at first angle is operated to capture a firstimage. Operation proceeds from step 4504 to step 4506, in which theangle of said light redirection device is changed to a second angle.Operation proceeds from step 4506 to step 4510. In step 4510, the firstoptical chain is operated to capture a second image while the lightredirection device is set at the second angle. Operation proceeds fromstep 4510 to step 4512.

In step 4512 a composite image is generated from the first and secondimages.

In some embodiments, the light redirection device is a mirror, e.g., asubstantially plane mirror. A substantially plane mirror includes aplane mirror which is used in some embodiments as well as mirrors whichmay have a slight curvature, e.g., mirrors with a curvature of in therange of 0 to 3 degrees in some embodiments but other ranges are alsopossible. In some such embodiments, the substantially plane mirror is aplane mirror. In various embodiments, the light redirection device is aprism. In some embodiments, step 4506 includes step 4508 in which anangle of the light redirection device is changed relative to at leastone of a plane of an image sensor used to capture the first and secondimages or a lens in the first optical chain.

In some embodiments, the first and second images are captured during afirst period of time, and step 4516 is performed. In step 4516 theposition of the camera device is maintained through the first period oftime so that the first and second images are captured from the samecamera device position. In some such embodiments, the first and secondimages correspond to the first and second portions of a scene area ofinterest, respectively, said first and second portions being at leastpartially non-overlapping.

In some embodiments, the first image includes a first scene areaportion, and the second image includes a second scene area portion, andthe first and second scene area portions are non-overlapping. In someembodiments, step 4512 includes step 4514. In step 4514 at least part ofsaid first image corresponding to a first scene area portion and atleast part of said second image corresponding to said second scene areaportion are incorporated into said composite image.

Thus, in this exemplary embodiment, the composite image is generated atleast from the non-overlapping image portions; the scene area to whichthe composite image corresponds is not a subset of image area of asingle one of the first and second images. In some embodiments, thefirst and second images may also include overlapping scene areaportions.

In some embodiments, step 4518 is performed in which light redirectiondevice movement is controlled as a function of video frame capture rate.In some such embodiments, step 4518 includes step 4520. In step 4520 thelight redirection device is moved so the z different images are capturedper video frame, where z corresponds to the number of images captured bythe first optical chain to generate a single composite video frame.

In some embodiments, the camera includes at least two optical chains andsteps 4522, 4524, and 4526 are performed. In step 4522 a second opticalchain of the camera device is operated to capture a third image, while asecond light redirection device, e.g., a mirror or a prism, in thesecond optical chain is at third angle. Operation proceeds from step4522 to step 4524. In step 4524 the angle of the second lightredirection device is set to a fourth angle. Operation proceeds fromstep 4524 to step 4526. In step 4526 the second optical chain isoperated to capture a fourth image while the second light redirectiondevice is set at the fourth angle.

In some embodiments, the first and third images correspond to a firstcapture time and the second and fourth images correspond to a secondcapture time. In some such embodiments, the camera is maintained in thesame position for the first and second capture times.

Operation proceeds from step 4526 to step 4512. In some embodiments,step 4512 includes step 4528 in which the composite image is generatedfrom first, second, third, and fourth images.

FIG. 48A is a drawing of an exemplary camera device 4600 in accordancewith an exemplary embodiment. Camera device 4600 includes a processor4602, e.g., a CPU, memory 4604, an assembly of modules 4606, e.g., anassembly of hardware modules such as circuits, a user input interface4608, a display 4610, an I/O interface 4612, and a plurality of opticalchains (optical chain 1 4618, . . . , optical chain N 4620) coupledtogether via a bus 4621 over which the various elements may interchangedata and information. Display 4610, e.g., a liquid crystal display(LCD), a light emitting diode (LED) display, an organic light emittingdiode (OLED) display or an active matrix organic light emitting diode(AMOLED) display, is for displaying images, image portions, user menus,etc., to the user of camera device 4600. User input interface 4608includes, e.g., switches, buttons, touch screen interface, etc., forreceiving input from the user. I/O interface 4612 includes a pluralityof communications ports for communicating with other devices, e.g., apersonal computer. I/O interface 4612 in some embodiments, includes oneor more wireless communications interfaces. Information communicatedover I/O interface 4612 includes, e.g., a combined images generated fromcaptured images from multiple optical chains, and/or captured imagesfrom individual optical chains to be processed and/or combinedexternally, e.g., in a personal computer.

Memory 4604 includes assembly of modules 4614 and data/information 4616.

Optical chain 1 4618 includes a lens 4622, e.g., an outer lens, amoveable mirror 4624, e.g., a moveable substantially plane mirror, andan image sensor 4626. In some embodiments, the substantially planemirror is a plane mirror. In various embodiments, the plane mirror has aflat light reflective surface. In some embodiments, the substantiallyplane mirror has a curvature in the range of 0 to 5 degrees. In variousembodiments, the moveable mirror 4624 includes a hinge mount. Opticalchain 1 4618 further includes a mirror movement module 4628, e.g., adrive motor and corresponding drive circuitry, and a mirror positiondetection module 4630, e.g., a angular position sensing device such as aresolver or synchro or optical rotary encoder or a magnetic rotaryencoder, and, in some embodiments, circuitry to convert the sensormeasurement into a digital value, e.g., an analog to digital converter.Image sensor 4626 is used to capture first and second images. Lightpasses through lens 4622 before reaching mirror 4624. Mirror 4624, whichis a light redirection device, directs the light which has passedthrough lens 4622 onto the image sensor 4626.

Optical chain N 4620 includes a lens 4632, e.g., an outer lens, amoveable mirror 4634, e.g., a moveable substantially plane mirror, andan image sensor 4636. In some embodiments, the moveable substantiallyplane mirror is a plane mirror. In various embodiments, the moveablemirror 4634 includes a hinge mount. Optical chain N 4620 furtherincludes a mirror movement module 4638, e.g., a drive motor andcorresponding drive circuitry, and a mirror position detection module4640, e.g., a angular position sensing device such as a resolver orsynchro or an optical rotary encoder or a magnetic rotary encoder, and,in some embodiments, circuitry to convert the sensor measurement into adigital value. One or more of the optical chain modules (4618, . . . ,4620) in some embodiments, include one or more additional elements,e.g., an additional lens, an additional mirror, a prism, a filter, etc.

FIG. 48B is a drawing of an exemplary camera device 4700 in accordancewith an exemplary embodiment. Camera device 4700 includes a processor4602, e.g., a CPU, memory 4604, an assembly of modules 4606, e.g., anassembly of hardware modules such as circuits, a user input interface4608, a display 4610, an I/O interface 4612, and a plurality of opticalchains (optical chain 1 4718, . . . , optical chain N 4720) coupledtogether via a bus 4621 over which the various elements may interchangedata and information. Display 4610, e.g., a liquid crystal display(LCD), a light emitting diode (LED) display, an organic light emittingdiode (OLED) display or an active matrix organic light emitting diode(AMOLED) display, is for displaying images, image portions, user menus,etc., to the user of camera device 4700. User input interface 4608includes, e.g., switches, buttons, touch screen interface, etc. forreceiving input from the user. I/O interface 4612 includes a pluralityof communications ports for communicating with other devices, e.g., apersonal computer. I/O interface 4612 in some embodiments, includes oneor more wireless communications interfaces. Information communicatedover I/O interface 4612 includes, e.g., a combined images generated fromcaptured images from multiple optical chains, and/or captured imagesfrom individual optical chains to be processed and/or combinedexternally, e.g., in a personal computer.

Memory 4604 includes assembly of modules 4614 and data/information 4616.

Optical chain 1 4718 includes a lens 4722, e.g., an outer lens, a prism4724, and an image sensor 4726. In various embodiments, the prism 4724includes a hinge mount. Optical chain 1 4718 further includes a prismmovement module 4728, e.g., a drive motor and corresponding drivecircuitry, and a prism position detection module 4730, e.g., a angularposition sensing device such as a resolver or synchro or optical rotaryencoder or a magnetic rotary encoder, and, in some embodiments,circuitry to convert the sensor measurement into a digital value, e.g.,an analog to digital converter. Image sensor 4726 is used to capturefirst and second images. Light passes through lens 4722 before reachingprism 4724. Prism 4724, which is a light redirection device, directs thelight which has passed through lens 4722 onto the image sensor 4726.

Optical chain N 4720 includes a lens 4732, e.g., an outer lens, amoveable prism 4734, and an image sensor 4736. In various embodiments,the moveable prism 4734 includes a hinge mount. Optical chain N 4720further includes a prism movement module 4738, e.g., a drive motor andcorresponding drive circuitry, and a prism position detection module4740, e.g., a angular position sensing device such as a resolver orsynchro or an optical rotary encoder or a magnetic rotary encoder, and,in some embodiments, circuitry to convert the sensor measurement into adigital value. One or more of the optical chain modules (4718, . . . ,4720) in some embodiments, include one or more additional elements,e.g., an additional lens, an additional prism, a mirror, a filter, etc.

FIG. 48 is a drawing of an assembly of modules 4800, which may beincluded in a camera device implemented in accordance with the presentinvention, e.g., camera 4600 of FIG. 46 or camera 4700 of FIG. 47.Assembly of modules 4800 may implement steps of a method, e.g., steps ofthe method of flowchart 4500 of FIG. 45. In some embodiments, assemblyof modules 4800 is an assembly of circuits, which may be coupledtogether. In one exemplary embodiment, assembly of modules 4800 isassembly of modules 4606 of camera 4600 of FIG. 46 or camera 4700 ofFIG. 47. In some embodiments, the assembly of module 4800 is an assemblyof software modules. In one exemplary embodiment, assembly of modules4800 is assembly of modules 4614 of memory 4604 of camera 4600 of FIG.46 or camera 4700 of FIG. 47.

FIG. 48C illustrates an assembly of modules 4800 which can, and in someembodiments is, used in the camera device 4600 illustrated in FIG. 48Aor camera device 4700 of FIG. 48B. The modules in the assembly ofmodules 4800 can, and in some embodiments are, implemented fully inhardware within the processor 4602, e.g., as individual circuits. Themodules in the assembly of modules 4800 can, and in some embodimentsare, implemented fully in hardware within the assembly of modules 4606,e.g., as individual circuits corresponding to the different modules. Inother embodiments some of the modules are implemented, e.g., ascircuits, within the processor 4602 with other modules beingimplemented, e.g., as circuits within assembly of modules 4606, externalto and coupled to the processor. As should be appreciated the level ofintegration of modules on the processor and/or with some modules beingexternal to the processor may be one of design choice.

Alternatively, rather than being implemented as circuits, all or some ofthe modules in assembly of modules 4800 may be implemented in softwareand stored in the memory 4604 of the camera device 4600 or camera device4700, with the modules controlling operation of camera device 4600 orcamera device 4700 to implement the functions corresponding to themodules when the modules are executed by a processor, e.g., processor4602. In some such embodiments, the assembly of modules 4800 is includedin the memory 4604 as assembly of modules 4614. In still otherembodiments, various modules in assembly of modules 4800 are implementedas a combination of hardware and software, e.g., with another circuitexternal to the processor providing input to the processor 4602 whichthen under software control operates to perform a portion of a module'sfunction. While shown in the FIG. 48A and FIG. 48B embodiments as asingle processor, e.g., computer, it should be appreciated that theprocessor 4602 may be implemented as one or more processors, e.g.,computers.

When implemented in software the modules include code, which whenexecuted by the processor 4602, configure the processor 4602 toimplement the function corresponding to the module. In embodiments wherethe assembly of modules 4800 is stored in the memory 4604, the memory4604 is a computer program product comprising a computer readable mediumcomprising code, e.g., individual code for each module, for causing atleast one computer, e.g., processor 4602, to implement the functions towhich the modules correspond

Completely hardware based or completely software based modules may beused. However, it should be appreciated that any combination of softwareand hardware, e.g., circuit implemented modules may be used to implementthe functions. As should be appreciated, the modules illustrated in FIG.48C control and/or configure the camera device 4600 or camera device4700 or elements therein such as the processor 4602, to performfunctions of the corresponding steps illustrated in the method flowchart4500 of FIG. 47. Thus the assembly of modules 4800 includes variousmodules that perform functions of the corresponding steps of the methodshown in FIG. 47.

FIG. 48C is an assembly of modules 4800 in accordance with an exemplaryembodiment. Assembly of modules 4800 includes a light redirection devicecontrol module 4802, an image capture control module 4804, a compositeimage generation module 4806, a stability control module 4808 and alight redirection device movement timing control module 4810.

Light redirection setting control module 4802, e.g., a mirrorredirection setting control module or prism redirection setting controlmodule, is configured to determine at least one light redirection deviceangle position to be used by the first optical, e.g., optical chain 14618 or optical chain 1 4718. Image capture control module 4804 isconfigured to operate the first optical chain to capture a first imagewhile the first light redirection device, e.g., mirror 4624 or prism4724, is set, under the direction of the light redirection devicecontrol module 4802, at a first angle and to capture a second imagewhile the first light redirection device is set, under the direction ofthe light redirection device control module 4802 at a second angle.Composite image generation module 4806 is configured to generate acomposite image from the first and second images. In some embodiments,the light redirection device is a mirror, e.g., mirror 4624, which is aplane mirror or a substantially plane mirror. In some embodiments, thelight redirection device is a prism, e.g., prism 4724.

Light redirection device control module 4802 is further configured tocontrol the angle of the light redirection device, e.g., mirror 4624 orprism 4724, to change when the determined light redirection device angleis different from a current light redirection device angle. In someembodiments, the light redirection device is a mirror, e.g., mirror4624, is a substantially plane mirror, and the change involves alteringan angle of the mirror relative to at least one of a plane of the imagesensor, e.g., sensor 4626, or a lens, e.g., lens 4622, included in thefirst optical chain.

In various embodiments, the first and second images, e.g., from thefirst optical chain, optical chain 1 4618 or optical chain 1 4718, arecaptured during a first period of time, and stability control module4808 is configured to maintain image stability throughout the firstperiod of time, e.g., so that the first and second images are capturedfrom the same camera position. In some embodiments, the first and secondimages correspond to first and second portions of a scene of interest,respectively, and said first and second portions are at least partiallynon-overlapping.

In some embodiments, the first image includes a first scene areaportion, and the second image includes a second scene area portion, andthe first and second scene portions are non-overlapping. In variousembodiments, the first and second images may also include overlappingscene area portions. In some such embodiments, composite imagegeneration module 4806 is configured to generate a composite image fromthe first and second images by incorporating at least a part of saidfirst image corresponding to said first scene area portion and a part ofsaid second image corresponding to said second scene area portion intosaid composite image. Thus in various embodiments, the composite imageis generated at least from the non-overlapping portions.

Light redirection device movement timing control module 4810 isconfigured to control light redirection device movement as a function ofvideo frame capture rate. In some such embodiments, the lightredirection device movement timing control module 4810 controls lightredirection device movement as a function of the video capture rate tomove the light redirection device so that z different images arecaptured per video frame, where z corresponds to the number of imagescaptured by the first optical chain to generate a single compositeimage.

In various embodiments, the camera device including assembly of modules4800 includes a second optical chain, e.g., optical chain N 4620 oroptical chain N 4720, including a second light redirection device, e.g.,moveable mirror 4634 or moveable prism 4734, which can be set atdifferent angles, and light redirection device control module 4802 isfurther configured to determine a third angle at which the second lightredirection device is to be positioned. In some such embodiments, theimage capture control module 4804 is further configured to control thesecond optical chain, e.g., module 4620 or module 4720, to capture athird image while the second light redirection device is set at a thirdangle, and capture a fourth image while the second light redirectiondevice is set at a fourth angle. In some such embodiments, the compositeimage generation module 4806 is further configured to generate thecomposite image from the third and fourth images, in addition to thefirst and second images. In some embodiments, the first and third imagescorrespond to a first capture time and the second and fourth imagescorrespond to a second capture time. In various embodiments, the cameradevice is maintained in the same position for the first and secondcapture times.

FIG. 49 is a drawing 4900 used to illustrate exemplary captured imagesused to generate a combined image in accordance with an exemplaryembodiment. Drawing 4900 includes an exemplary camera device 4902including two optical chains in accordance with an exemplary embodiment.Camera device 4900 is, e.g., camera device 4600 of FIG. 48A or cameradevice 4700 of FIG. 48B, a camera device including assembly of modules4800 of FIG. 48C, and/or a camera device implementing the method offlowchart 4500 of FIG. 47. Each of the optical chains of camera device4900 includes and a lens, e.g., an outer lens, a controllable moveablelight redirection device, e.g., a moveable mirror or a moveable prism,and an image sensor. In one example, lens L1 4904 is the outer lens foroptical chain 1, and lens L2 4906 is the outer lens for optical chain 2.

Drawing 4908 of FIG. 49 illustrates a first captured image 4912 within ascene of interest 4910. Drawing 4914 of FIG. 49 illustrates a secondcaptured image 4916 within the scene of interest 4910. Drawing 4918 ofFIG. 49 illustrates a third captured image 4920 within the scene ofinterest 4910. Drawing 4922 of FIG. 49 illustrates a fourth capturedimage 4924 within the scene of interest 4910. Note that each of theimages (first image 4912, second image 4916, third image 4920, fourthimage 4924) includes a non-overlapping image portion with respect to theother images. Drawing 4944 illustrates a composite image 4925,corresponding to the scene area of interest 4910, which is generatedfrom the captured images (first image 4912, second image 4916, thirdimage 4920 and fourth image 4924).

Table 4926 illustrates which optical chains are used to capture whichimages, exemplary light redirection device angle settings used for imagecapture, and exemplary capture times corresponding to the capturedimages. First column 4928 identifies the captured image. Second column4930 identifies the optical chain used to capture the identified imagein the same row of the first column. Third column 4932 identifies anexemplary light redirection device angle setting to which the lightredirection device of the optical chain which captured the image was setat the time of image capture. Fourth column 4934 identifies an exemplarycapture time during which the corresponding image of the same row wascaptured.

Row 4936 identifies that first image 4912 was captured by optical chain1; the controllable moveable light redirection device of optical chain 1was set at a first angle when the first image was captured; and thefirst image was captured during a first capture time interval. Row 4938identifies that second image 4916 was captured by optical chain 1; thecontrollable moveable light redirection device of optical chain 1 wasset at a second angle when the second image was captured; and the secondimage was captured during a second capture time interval. Row 4940identifies that third image 4920 was captured by optical chain 2; thecontrollable moveable light redirection device of optical chain 2 wasset at a third angle when the third image was captured; and the thirdimage was captured during a first capture time interval. Row 4942identifies that fourth image 4924 was captured by optical chain 2; thecontrollable moveable light redirection device of optical chain 2 wasset at a fourth angle when the fourth image was captured; and the fourthimage was captured during a second capture time interval.

In one exemplary embodiment, the second angle is greater than the firstangle, and the fourth angle is greater than the third angle. In somesuch embodiments, the first angle is substantially the same as the thirdangle, and the second angle is substantially the same as the fourthangle. In various embodiments, the first capture time is the sameduration as the second capture time. In some embodiments, the moveablelight redirection device of the first optical chain and the moveablelight redirection device of the second optical chain are moved inparallel during a time interval between first capture time and thesecond capture time, e.g., to facilitate a small time gap between thefirst capture time and second capture time. In some embodiments, themoveable light redirection device of the first optical chain and themoveable light redirection device of the second optical chain are movedserially during a time interval between first capture time and thesecond capture time, e.g., to limit instantaneous battery power drain.

FIG. 50 is a drawing 5000 used to illustrate exemplary captured imagesused to generate a combined image in accordance with an exemplaryembodiment. Drawing 5000 includes an exemplary camera device 5002including three optical chains in accordance with an exemplaryembodiment. Camera device 5000 is, e.g., camera device 4600 of FIG. 48A,camera 4700 of FIG. 48B, a camera device including assembly of modules4800 of FIG. 48C, and/or a camera device implementing the method offlowchart 4500 of FIG. 47. Each of the optical chains of camera device5000 includes a lens, e.g., an outer lens, a controllable moveable lightredirection device, and an image sensor. Lens L1 5004 is the outer lensfor optical chain 1, lens L2 5006 is the outer lens for optical chain 2,and lens L3 5008 is the outer lens for optical chain 3.

Drawing 5010 of FIG. 50 illustrates captured image I1 5012 within ascene of interest 5011. Drawing 5013 of FIG. 50 illustrates capturedimage I2 5014 within the scene of interest 5011. Drawing 5015 of FIG. 50illustrates captured image I3 5016 within the scene of interest 5011.Drawing 5017 of FIG. 50 illustrates captured image I4 5018 within thescene of interest 5011. Drawing 5019 of FIG. 50 illustrates capturedimage I5 5020 within the scene of interest 5011. Drawing 5021 of FIG. 50illustrates captured image I6 5022 within the scene of interest 5011.Drawing 5023 of FIG. 50 illustrates captured image I7 5026 within thescene of interest 5011. Drawing 5025 of FIG. 50 illustrates capturedimage I8 5026 within the scene of interest 5011. Drawing 5027 of FIG. 50illustrates captured image I9 5028 within the scene of interest 5011.Drawing 5029 of FIG. 50 illustrates captured image I10 5030 within thescene of interest 5011. Drawing 5031 of FIG. 50 illustrates capturedimage I11 5033 within the scene of interest 5011. Drawing 5034 of FIG.50 illustrates captured image I12 5035 within the scene of interest5011.

A composite image 5060, corresponding to the scene area of interest 5011is generated from the captured images (I1 5012, I2 5014, I3 5016, I45018, I5 5020, I6 5022, I7 5024, I8 5026, I9 5028, I10 5030, I11 5033,I12 5035).

Table 5036 illustrates which optical chains of camera 5002 are used tocapture which images, exemplary light redirection device angle settingsused for image capture, and exemplary capture times corresponding to thecaptured images. First column 5037 identifies the captured image. Secondcolumn 5038 identifies the optical module used to capture the identifiedimage in the same row of the first column. Third column 5039 identifiesan exemplary light redirection device angle setting to which themoveable light redirection device of the optical chain which capturedthe image was set at the time of image capture. Fourth column 5050identifies an exemplary capture time during which the correspondingimage of the same row was captured.

The second row of table 5036 identifies that image I1 5012 was capturedby optical chain 1; the controllable light redirection device of opticalchain 1 was set at a angle α=α1 when image I1 was captured; and image I1was captured during a first capture time interval. The third row oftable 5036 identifies that image I2 5014 was captured by optical chain1; the controllable moveable light redirection device of optical chain 1was set at an angle α=α2 when the image I2 was captured; and the imageI2 was captured during a second capture time interval. The fourth row oftable 5036 identifies that image I3 5016 was captured by optical chain1; the controllable light redirection device of optical chain 1 was setat an angle α=α3 when the image I3 5016 was captured; and image I3 5016was captured during a third capture time interval. The fifth row oftable 5036 identifies that image I4 5018 was captured by optical chain1; the controllable moveable light redirection device of optical chain 1was set at an angle α=α4 when image I4 was captured; and image I4 wascaptured during a fourth capture time interval. In some embodiments,α4>α3>α2>α1. In some other embodiments, α4<α3<α2<α1.

The sixth row of table 5036 identifies that image I5 5020 was capturedby optical chain 2; the controllable moveable light redirection deviceof optical chain 2 was set at an angle β=β1 when image I5 5020 wascaptured; and image I5 was captured during a first capture timeinterval. The seventh row of table 5036 identifies that image I6 5022was captured by optical chain 2; the controllable moveable lightredirection device of optical chain 2 was set at an angle β=β2 whenimage I6 was captured; and image I6 was captured during a second capturetime interval. The eighth row of table 5036 identifies that image I75024 was captured by optical chain 2; the controllable moveable lightredirection device of optical chain 2 was set at an angle β=β3 whenimage I7 5024 was captured; and image I7 5024 was captured during athird capture time interval. The ninth row of table 5036 identifies thatimage I8 5026 was captured by optical chain 2; the controllable moveablelight redirection device of optical chain 2 was set at an angle β=β4when image I8 5026 was captured; and image I8 5026 was captured during afourth capture time interval. In some embodiments, β4>β3>β2>β1. In someembodiments, β4<β3<β2<β1.

The tenth row of table 5036 identifies that image I9 5028 was capturedby optical chain 3; the controllable moveable light redirection deviceof optical chain 3 was set at a angle θ=θ1 when image I9 5028 wascaptured; and image I9 was captured during a first capture timeinterval. The eleventh row of table 5036 identifies that image I10 5030was captured by optical chain 3; the controllable moveable lightredirection device of optical chain 3 was set at an angle θ=θ2 whenimage I10 was captured; and image I10 was captured during a secondcapture time interval. The twelfth row of table 5036 identifies thatimage I11 5033 was captured by optical chain 3; the controllablemoveable light redirection device of optical chain 3 was set at an angleθ=θ3 when image I11 5033 was captured; and image I11 5033 was capturedduring a third capture time interval. The thirteenth row of table 5036identifies that image I12 5026 was captured by optical chain 3; thecontrollable moveable light redirection device of optical chain 3 wasset at an angle θ=θ4 when image I12 5035 was captured; and image I125035 was captured during a fourth capture time interval. In someembodiments, θ4>θ3>θ2>θ1. In some embodiments, θ4<θ3<θ2<θ1.

FIG. 51, comprising the combination of FIG. 51A and FIG. 51B, is aflowchart 5100 of an exemplary method of capturing at least one image.The exemplary method of flowchart 5100 may be performed by a cameradevice including five or more optical chains or by a camera deviceincluding five or more optical chains and an image processing device. Insome embodiments, the camera device is one of: a smart phone and acamera. In some embodiments, the image processing device is a personalcomputer.

Operation of the exemplary method starts in step 5102 and proceeds tostep 5104. In step 5104, information mapping zoom settings to lightredirection device positions is stored in memory. In variousembodiments, the light redirection devices are mirrors, e.g., planemirror, or prisms. Operation proceeds from step 5104 to step 5106. Instep 5106 user input indicating a zoom setting is received. Operationproceeds from step 5106 to step 5108. In step 5108 light redirectiondevice positions, based on the indicated zoom setting, are computedprior to moving a first light redirection device in accordance with theindicated zoom setting. Thus, in step 5108 light redirection devicepositions for moveable controllable light redirection devices aredetermined corresponding to a zoom setting, e.g., from a stored look-uptable in memory. For example, in step 5108 the camera device determines:a first position for a first light redirection device of a first opticalchain corresponding to a the indicated zoom setting, a second positionfor a second light redirection device of a third optical chaincorresponding to a the indicated zoom setting, a third position for athird light redirection device of a fourth optical chain correspondingto a the indicated zoom setting, and a fourth position for a fourthlight redirection device of a fifth optical chain corresponding to theindicated zoom setting. In some embodiments, corresponding to each of aplurality of predetermined discrete possible alternative user selectablezoom settings, there is a corresponding set of light redirection devicepositions and/or angles for each of the moveable light redirectiondevices. Operation proceeds from step 5108 to steps 5110, 5112, 5114,and 5116, which may be performed in parallel.

In step 5110 a first light redirection device, said first lightredirection device being one of a substantially plane mirror or a prism,of a first optical chain or the camera device is moved to a firstposition in accordance with the indicated zoom setting. In someembodiments, the substantially plane mirror is a plane mirror. In someembodiments, the first light redirection device is a plane mirror, andsaid first position is a first mirror position corresponding to a firstdirection of the first plane mirror. In various embodiments, the firstdirection of the first plane mirror is defined as the direction of theperpendicular to the plane of the first plane mirror. In step 5112 asecond light redirection device of a third optical chain of the cameradevice is moved to a second position in accordance with the indicatedzoom setting. In some embodiments, the second light redirection deviceis a second plane mirror, and said second position is a second mirrorposition corresponding to a first direction of the second plane mirror.In various embodiments, the first direction of the second plane mirroris defined as the direction of the perpendicular to the plane of thesecond plane mirror. In step 5114 a third light redirection device of afourth optical chain of the camera device is moved to a third positionin accordance with the indicated zoom setting. In some embodiments, thethird light redirection device is a third plane mirror, and said thirdposition is a third mirror position corresponding to a first directionof the third plane mirror. In various embodiments, the first directionof the third plane mirror is defined as the direction of theperpendicular to the plane of the third plane mirror. In step 5116 alight redirection device of a fifth optical chain of the camera deviceis moved to a fourth position in accordance with the indicated zoomsetting. In some embodiments, said fourth light redirection device is afourth plane mirror, and said fourth position is a fourth mirrorposition corresponding to a first direction of the fourth plane mirror.In various embodiments, the first direction of the fourth plane mirroris defined as the direction of the perpendicular to the plane of thefourth plane mirror. Operation proceeds from steps 5110, 5112, 5114, and5116, via connecting node A 5118, to steps 5120, 5122, 5124, 5126 and5128, and in some embodiments to steps 5130 and 5132. In variousembodiments, steps 5120, 5122, 5124, 5126, 5128, 5130 and 5132 areperformed in parallel.

In step 5120 the camera device captures a first image of a first portionof a scene area of interest using said first optical chain while saidfirst light redirection device is at the first position. In step 5122,the camera device captures a second image of a second portion of thescene area of interest using the second optical chain, while said firstlight redirection device is at the first position. In some embodiments,the first image is captured during a period of time which at leastpartially overlaps the period of time during which the second image iscaptured. In some embodiments, said first image is captured at the sametime said second image is captured. In step 5124 the camera devicecaptures a third image of a third portion of the scene area of interestusing a third optical chain while said first light redirection device isat the first position. In step 5126 the camera device captures a fourthimage of a fourth portion of the scene area of interest using saidfourth optical chain while said first light redirection device is at thefirst position. In step 5128 the camera device captures a fifth image ofa fifth portion of the scene area of interest using said fifth opticalchain while said first light redirection device is at the firstposition. In some embodiments, one or both of optional step 5130 andoptional step 5132 are performed. In step 5130, the camera devicecaptures a sixth image of a sixth portion of the scene area of interestusing a sixth optical chain, while said first light redirection deviceis at the first position. In step 5132, the camera device captures aseventh image of a seventh portion of the scene area of interest using aseventh optical chain, while said first light redirection device is atthe first position.

In various embodiments, the first, second, third and fourth plane lightredirection devices are not being moved during the image captureoperations of steps 5120, 5122, 5124, 5126 and 5128. In variousembodiments, the first, second, third and fourth light redirectiondevices are not being moved during the image capture operations of steps5120, 5122, 5124, 5126, 5128, 5130 and 5132. In various embodiments, thefirst, second, third, fourth and fifth optical chains capture the imagesat the same time, e.g., when set for a particular zoom setting. Invarious embodiments, the first, second, third, fourth, fifth, sixth andseventh optical chains capture the images at the same time, e.g., whenset for a particular zoom setting.

Operation proceeds from steps 5120, 5122, 5124, 5126, 5128, and in someembodiments, steps 5130 and/or 5132 to step 5134. In step 5134 thecaptured images are combined to generate a composite image correspondingto the indicated zoom setting. In some embodiments, step 5134 includesone or both of steps 5136 and 5138. In step 5136 the first, second,third, fourth, and fifth images are combined to generate a compositeimage corresponding to the indicated zoom setting. In step 5138 thefirst, second, third, fourth, fifth, sixth and seventh images arecombined to generate a composite image corresponding to the indicatedzoom setting. Operation proceeds from step 5134 to step 5140, in whichthe generated composite image is stored in memory. Steps 5134 and 5136may be performed by the camera device or by an image processing device.Operation proceeds from step 5140, via connecting node B 5142 to step5106.

In some embodiments, the first and second optical chains have differentfocal lengths. In some such embodiments, the first optical chain has alarger focal length than the second optical chain. In some suchembodiments, the second optical chain has a smaller focal length thaneach of the first, second third and fourth optical chains. In someembodiments, the first, third, fourth and fifth optical chains have thesame focal length and capture smaller portions of the scene area ofinterest than the second optical chain.

In various embodiments, the first optical chain includes a first lightredirection device which is a moveable light redirection device and thesecond optical chain does not include a moveable light redirectiondevice.

In various embodiments, the first, third, fourth, and fifth opticalchains each include a moveable controllable light redirection device,and the second, sixth and seventh optical chains do not include amoveable controllable light redirection device, e.g., the lightredirection devices in the second, sixth and seventh optical chains arefixed position light redirection devices, e.g., each of which is hardmounted in a fixed position. In some embodiments, two of the second,sixth, and seventh images are substantially the same size. In someembodiments, one of the second, sixth, and seventh images issubstantially the same size as each of the first, third, fourth, andfifth images.

FIG. 52 is a drawing of an exemplary camera device 5200 in accordancewith an exemplary embodiment. Camera device 5200 includes a processor5202, e.g., a CPU, memory 5204, an assembly of modules 5206, e.g., anassembly of hardware modules such as circuits, a user input interface5208, a display 5210, an I/O interface 5212, and a plurality of opticalchains (optical chain 1 5218, optical chain 2 5220, optical chain 35222, optical chain 4 5224, optical chain 5 5226, optical chain 6 5228,optical chain 7 5230) coupled together via a bus 5231 over which thevarious elements may interchange data and information. In someembodiments, optical chain 6 5228 and optical chain 7 5230 are optional.Display 5210, e.g., a liquid crystal display (LCD), a light emittingdiode (LED) display, an organic light emitting diode (OLED) display oran active matrix organic light emitting diode (AMOLED) display, is fordisplaying images, image portions, user menus, etc., to the user ofcamera 5200. User input interface 5208 includes, e.g., switches,buttons, touch screen interface, etc. for receiving input from the user.I/O interface 5212 includes a plurality of communications ports forcommunicating with other devices, e.g., a personal computer. I/Ointerface 5212 in some embodiments, includes one or more wirelesscommunications interfaces. Information communicated over I/O interface5212 includes, e.g., combined images generated from captured images frommultiple optical chains, and/or captured images from individual opticalchains to be processed and/or combined externally, e.g., in a personalcomputer.

Memory 5204 includes assembly of modules 5214 and data/information 5216.Data/information 5216, in some embodiments, includes stored informationmapping zoom settings to light redirection device positions, e.g., alook-up table 5500 of FIG. 55.

Optical chain 1 5218 includes a lens 5232, e.g., an outer lens, amoveable substantially plane mirror 5234, and an image sensor 5236. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 5234 includes a hinge mount.Optical chain 1 5218 further includes a mirror movement module 5238,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 5240, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 5236 is used to capture a first image of a firstportion of a scene area of interest. Light passes through lens 5232before reaching mirror 5234. Mirror 5234, which is a light redirectiondevice, directs the light which has passed through lens 5232 onto theimage sensor 5234.

Optical chain 2 5220 includes a lens 5242, e.g., an outer lens, a fixedmirror 5244, e.g., a substantially plane fixed mirror, and an imagesensor 5246. In some embodiments, the substantially plane fixed mirroris a fixed plane mirror. Image sensor 5246 is used to capture a secondimage of a second portion of a scene area of interest. Light passesthrough lens 5242 before reaching mirror 5246. Mirror 5244, which is alight redirection device, directs the light which has passed throughlens 5242 onto the image sensor 5246.

Optical chain 3 5222 includes a lens 5248, e.g., an outer lens, amoveable substantially plane mirror 5250, and an image sensor 5252. Invarious embodiments, the moveable mirror 5250 includes a hinge mount.Optical chain 3 5222 further includes a mirror movement module 5254,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 5256, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 5252 is used to capture a third image of a thirdportion of a scene area of interest. Light passes through lens 5248before reaching mirror 5250. Mirror 5250, which is a light redirectiondevice, directs the light which has passed through lens 5248 onto theimage sensor 5252.

Optical chain 4 5224 includes a lens 5258, e.g., an outer lens, amoveable substantially plane mirror 5260, and an image sensor 5262. Invarious embodiments, the moveable mirror 5260 includes a hinge mount.Optical chain 4 5224 further includes a mirror movement module 5264,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 5266, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 5262 is used to capture a fourth image of afourth portion of a scene area of interest. Light passes through lens5258 before reaching mirror 5260. Mirror 5260, which is a lightredirection device, directs the light which has passed through lens 5258onto the image sensor 5262.

Optical chain 5 5226 includes a lens 5268, e.g., an outer lens, amoveable substantially plane mirror 5270, and an image sensor 5272. Invarious embodiments, the moveable mirror 5270 includes a hinge mount.Optical chain 5 5226 further includes a mirror movement module 5274,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 5276, e.g., a angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 5272 is used to capture a fifth image of a firthportion of a scene area of interest. Light passes through lens 5268before reaching mirror 5270. Mirror 5270, which is a light redirectiondevice, directs the light which has passed through lens 5268 onto theimage sensor 5272.

Optical chain 6 5228 includes a lens 5278, e.g., an outer lens, a fixedmirror 5280, e.g., a fixed substantially plane mirror, and an imagesensor 5282. In some embodiments, the fixed substantially plane mirroris a fixed plane mirror. Image sensor 5282 is used to capture a sixthimage of a sixth portion of a scene area of interest. Light passesthrough lens 5278 before reaching mirror 5280. Mirror 5280, which is alight redirection device, directs the light which has passed throughlens 5278 onto the image sensor 5282.

Optical chain 7 5230 includes a lens 5284, e.g., an outer lens, a fixedmirror 5286, e.g., a fixed substantially plane mirror, and an imagesensor 5288. In some embodiments, the fixed substantially plane mirroris a fixed plane mirror. In an alternative embodiment, optical chain 75230 includes a moveable substantially plane mirror which is set to afixed position, e.g., irrespective of the zoom setting. Image sensor5288 is used to capture a seventh image of a seventh portion of a scenearea of interest. Light passes through lens 5284 before reaching mirror5286. Mirror 5286, which is a light redirection device, directs thelight which has passed through lens 5284 onto the image sensor 5288.

In some exemplary embodiments, the size of the second image issubstantially the same as the size of the scene or interest. In someembodiments, the size of the first image, third image, fourth image, andfifth image are substantially the same. In some such embodiments, thesize of the first image is approximately ¼ the size of the scene ofinterest. In some embodiments, the size of the seventh image issubstantially the same as the size of the first image, and the size ofthe sixth image is substantially the same as the size of the secondimage. In various embodiments, the second image and the sixth image areused to compute depth information.

FIG. 53 is a drawing of an exemplary camera device 5300 in accordancewith an exemplary embodiment. Camera device 5300 includes a processor5202, e.g., a CPU, memory 5204, an assembly of modules 5206, e.g., anassembly of hardware modules such as circuits, a user input interface5208, a display 5210, an I/O interface 5212, and a plurality of opticalchains (optical chain 1 5318, optical chain 2 5320, optical chain 35322, optical chain 4 5324, optical chain 5 5326, optical chain 6 5328,optical chain 7 5330) coupled together via a bus 5231 over which thevarious elements may interchange data and information. In someembodiments, optical chain 6 5328 and optical chain 7 5330 are optional.Display 5210, e.g., a liquid crystal display (LCD), a light emittingdiode (LED) display, an organic light emitting diode (OLED) display oran active matrix organic light emitting diode (AMOLED) display, is fordisplaying images, image portions, user menus, etc., to the user ofcamera 5300. User input interface 5208 includes, e.g., switches,buttons, touch screen interface, etc. for receiving input from the user.I/O interface 5212 includes a plurality of communications ports forcommunicating with other devices, e.g., a personal computer. I/Ointerface 5212 in some embodiments, includes one or more wirelesscommunications interfaces. Information communicated over I/O interface5212 includes, e.g., combined images generated from captured images frommultiple optical chains, and/or captured images from individual opticalchains to be processed and/or combined externally, e.g., in a personalcomputer.

Memory 5204 includes assembly of modules 5214 and data/information 5216.Data/information 5216, in some embodiments, includes stored informationmapping zoom settings to light redirection device positions, e.g., alook-up table 5500 of FIG. 55.

Optical chain 1 5318 includes a lens 5332, e.g., an outer lens, amoveable prism 5334 and an image sensor 5336. In various embodiments,the moveable prism 5334 includes a hinge mount. Optical chain 1 5318further includes a prism movement module 5338, e.g., a drive motor andcorresponding drive circuitry, and a prism position detection module5340, e.g., a angular position sensing device such as a resolver orsynchro or optical rotary encoder or a magnetic rotary encoder, and, insome embodiments, circuitry to convert the sensor measurement into adigital value, e.g., an analog to digital converter. Image sensor 5336is used to capture a first image of a first portion of a scene area ofinterest. Light passes through lens 5332 before reaching prism 5334.Prism 5334, which is a light redirection device, directs the light whichhas passed through lens 5332 onto the image sensor 5334.

Optical chain 2 5320 includes a lens 5342, e.g., an outer lens, a fixedprism 5344 and an image sensor 5346. Image sensor 5346 is used tocapture a second image of a second portion of a scene area of interest.Light passes through lens 5342 before reaching prism 5346. Prism 5344,which is a light redirection device, directs the light which has passedthrough lens 5342 onto the image sensor 5346.

Optical chain 3 5322 includes a lens 5348, e.g., an outer lens, amoveable prism 5350 and an image sensor 5352. In various embodiments,the moveable prism 5350 includes a hinge mount. Optical chain 3 5322further includes a prism movement module 5354, e.g., a drive motor andcorresponding drive circuitry, and a prism position detection module5346, e.g., a angular position sensing device such as a resolver orsynchro or optical rotary encoder or a magnetic rotary encoder, and, insome embodiments, circuitry to convert the sensor measurement into adigital value, e.g., an analog to digital converter. Image sensor 5352is used to capture a third image of a third portion of a scene area ofinterest. Light passes through lens 5348 before reaching prism 5350.Prism 5350, which is a light redirection device, directs the light whichhas passed through lens 5348 onto the image sensor 5352.

Optical chain 4 5324 includes a lens 5358, e.g., an outer lens, amoveable prism 5360 and an image sensor 5362. In various embodiments,the moveable prism 5360 includes a hinge mount. Optical chain 4 5324further includes a prism movement module 5364, e.g., a drive motor andcorresponding drive circuitry, and a prism position detection module5366, e.g., a angular position sensing device such as a resolver orsynchro or optical rotary encoder or a magnetic rotary encoder, and, insome embodiments, circuitry to convert the sensor measurement into adigital value, e.g., an analog to digital converter. Image sensor 5362is used to capture a fourth image of a fourth portion of a scene area ofinterest. Light passes through lens 5358 before reaching prism 5360.Prism 5360, which is a light redirection device, directs the light whichhas passed through lens 5358 onto the image sensor 5362.

Optical chain 5 5326 includes a lens 5368, e.g., an outer lens, amoveable prism 5370 and an image sensor 5372. In various embodiments,the moveable prism 5370 includes a hinge mount. Optical chain 5 5326further includes a prism movement module 5374, e.g., a drive motor andcorresponding drive circuitry, and a prism position detection module5376, e.g., a angular position sensing device such as a resolver orsynchro or optical rotary encoder or a magnetic rotary encoder, and, insome embodiments, circuitry to convert the sensor measurement into adigital value, e.g., an analog to digital converter. Image sensor 5372is used to capture a fifth image of a fifth portion of a scene area ofinterest. Light passes through lens 5378 before reaching prism 5370.Prism 5370, which is a light redirection device, directs the light whichhas passed through lens 5368 onto the image sensor 5372.

Optical chain 6 5328 includes a lens 5278, e.g., an outer lens, a fixedprism 5380 and an image sensor 5382. Image sensor 5382 is used tocapture a sixth image of a sixth portion of a scene area of interest.Light passes through lens 5378 before reaching prism 5380. Prism 5380,which is a light redirection device, directs the light which has passedthrough lens 5378 onto the image sensor 5382.

Optical chain 7 5330 includes a lens 5384, e.g., an outer lens, a fixedprism 5386 and an image sensor 5388. In some embodiments, fixed positionprism 5386 of optical chain 7 5338 is replaced by a moveable prism whichis set at a fixed position irrespective of the zoom setting. Imagesensor 5388 is used to capture a seventh image of a seventh portion of ascene area of interest. Light passes through lens 5384 before reachingprism 5386. Prism 5386, which is a light redirection device, directs thelight which has passed through lens 5384 onto the image sensor 5388.

In some exemplary embodiments, the size of the second image issubstantially the same as the size of the scene or interest. In someembodiments, the size of the first image, third image, fourth image, andfifth image are substantially the same. In some such embodiments, thesize of the first image is approximately ¼ the size of the scene ofinterest. In some embodiments, the size of the seventh image issubstantially the same as the size of the first image, and the size ofthe sixth image is substantially the same as the size of the secondimage. In various embodiments, the second image and the sixth image areused to compute depth information.

FIG. 54 is a drawing of an assembly of modules 5400, which may beincluded in a camera device implemented in accordance with the presentinvention, e.g., camera 5200 of FIG. 52 or camera 5300 of FIG. 53.Assembly of modules 5400 may implement steps of a method, e.g., steps ofthe method of flowchart 5100 of FIG. 51. In some embodiments, assemblyof modules 5400 is an assembly of circuits, which may be coupledtogether. In one exemplary embodiment, assembly of modules 5400 isassembly of modules 5206 of camera 5200 of FIG. 52 or camera 5300 ofFIG. 53. In some embodiments, the assembly of module 5400 is an assemblyof software modules. In one exemplary embodiment, assembly of modules5400 is assembly of modules 5214 of memory 5204 of camera device 5200 ofFIG. 52 or camera device 5300 of FIG. 53.

FIG. 54 illustrates an assembly of modules 5400 which can, and in someembodiments is, used in the camera device 5200 illustrated in FIG. 52 orcamera device 5300 of FIG. 53. The modules in the assembly of modules5400 can, and in some embodiments are, implemented fully in hardwarewithin the processor 5202, e.g., as individual circuits. The modules inthe assembly of modules 5400 can, and in some embodiments are,implemented fully in hardware within the assembly of modules 5206, e.g.,as individual circuits corresponding to the different modules. In otherembodiments some of the modules are implemented, e.g., as circuits,within the processor 5202 with other modules being implemented, e.g., ascircuits within assembly of modules 5206, external to and coupled to theprocessor. As should be appreciated the level of integration of moduleson the processor and/or with some modules being external to theprocessor may be one of design choice.

Alternatively, rather than being implemented as circuits, all or some ofthe modules in assembly of modules 5400 may be implemented in softwareand stored in the memory 5204 of the camera device 5200 or camera device5300, with the modules controlling operation of camera device 5200 orcamera device 5300 to implement the functions corresponding to themodules when the modules are executed by a processor, e.g., processor5202. In some such embodiments, the assembly of modules 5400 is includedin the memory 5204 as assembly of modules 5214. In still otherembodiments, various modules in assembly of modules 5400 are implementedas a combination of hardware and software, e.g., with another circuitexternal to the processor providing input to the processor 5202 whichthen under software control operates to perform a portion of a module'sfunction. While shown in the FIG. 52 and FIG. 53 embodiments as a singleprocessor, e.g., computer, it should be appreciated that the processor5202 may be implemented as one or more processors, e.g., computers.

When implemented in software the modules include code, which whenexecuted by the processor 5202, configure the processor 5202 toimplement the function corresponding to the module. In embodiments wherethe assembly of modules 5400 is stored in the memory 5204, the memory5204 is a computer program product comprising a computer readable mediumcomprising code, e.g., individual code for each module, for causing atleast one computer, e.g., processor 5202, to implement the functions towhich the modules correspond.

Completely hardware based or completely software based modules may beused. However, it should be appreciated that any combination of softwareand hardware, e.g., circuit implemented modules may be used to implementthe functions. As should be appreciated, the modules illustrated in FIG.54 control and/or configure the camera device 5200 or camera device 5300or elements therein such as the processor 5202, to perform functions ofthe corresponding steps illustrated in the method flowchart 5100 of FIG.51. Thus the assembly of modules 5400 includes various modules thatperform functions of the corresponding steps of the method shown in FIG.51.

FIG. 54 is a drawing of an assembly of modules 5400 in accordance withan exemplary embodiment. Assembly of modules 5400 includes a lightredirection device positioning control module 5402, an image capturecontrol module 5404, a composite image generation module 5406, a lightredirection device position computation module 5408, and a user zoominput module 5410.

Light redirection device positioning control module 5402 is configuredto control a first light redirection device, included in a first opticalchain, to move to a first position in accordance with an indicated zoomsetting, said first light redirection device being one of asubstantially plane mirror or a prism. In some embodiments, thesubstantially plane mirror is a plane mirror.

Image capture control module 5404 is configured to control a firstoptical chain to capture a first image of a first portion of a scene ofinterest while said first light redirection device is at a firstposition. Image capture control module 5404 is further configured toControl a second optical chain to capture a second image of a secondportion of a scene of interest, the second image being captured whilesaid first light redirection device is at the first position. In someembodiments, the first and second images are captured at the same time,under the control of the image capture control module 5404.

In some embodiments, the first and second optical chains have differentfocal length. In some such embodiments, the first optical chain has alarger focal length than the second optical chain. In variousembodiments, the first light redirection device includes a movable lightredirection device and the second optical chain does not include amoveable light redirection device.

In various embodiments, the camera device including assembly of modules5400 includes a third optical chain including a second light redirectiondevice, a fourth optical chain including a third light redirectiondevice, and a fifth optical chain including a fourth light redirectiondevice. In some such embodiments, the light redirection devicepositioning control module 5402 is further configured to: move thesecond light redirection device of the third optical chain to a secondposition in accordance with an indicated zoom setting, move the thirdlight redirection device of the fourth optical chain to a third positionin accordance with an indicated zoom setting, and move the fourth lightredirection device of the fifth optical chain to a fourth position inaccordance with an indicated zoom setting. In some such embodiments, theimage capture control module 5404 is further configured to: control thethird optical chain to capture a third image of a third portion of thescene of interest while the first light redirection device is at thefirst position, control the fourth optical chain to capture a fourthimage of a fourth portion of the scene of interest while the first lightredirection device is at the first position, and control the fifthoptical chain to capture a fifth image of a fifth portion of the sceneof interest while the first light redirection device is at the firstposition.

In some embodiments, the camera device including assembly of modules5400 further includes a sixth optical chain and a seventh optical chain,and the image capture control module 5404 is further configured to:control the sixth optical chain to capture a sixth image of sixthportion of the scene of interest while the first light redirectiondevice is at the first position, and control the seventh optical chainto capture a seventh image of seventh portion of the scene of interestwhile the first light redirection device is at the first position. Invarious embodiments, the first, second, third, fourth, fifth, sixth, andseventh optical chains are controlled under the direction of the imagecapture control module 5404 to capture images during a time interval inwhich the moveable light redirection devices are not moving. In variousembodiments, the first, second, third, fourth, fifth, sixth, and seventhoptical chains are controlled under the direction of the image capturecontrol module 5404 to capture images such that each of the capturedimages at least partially overlaps the other captured images in time. Insome embodiments, the first, second, third, fourth, fifth, sixth, andseventh optical chains are controlled under the direction of the imagecapture control module 5404 to capture images at the same time.

Composite image generation module 5406 is configured to combine at leastthe first and second images to generate a composite image correspondingto the indicated zoom setting. In some embodiments, in which the cameradevice including assembly of modules 5400 includes five optical chains,the composite image generation module 5406 is configured to combine thefirst, second, third, fourth, and fifth images to generate a compositeimage corresponding to the indicated zoom setting. In some embodiments,in which the camera device including assembly of modules 5400 includesseven optical chains, the composite image generation module 5406 isconfigured to combine the first, second, third, fourth, fifth, sixth,and seventh images to generate a composite image corresponding to theindicated zoom setting, e.g., combine information from overlappingportions.

In various embodiments, the first, third, fourth, and fifth opticalchains have the same focal length and capture smaller portions of thescene area of interest than the second optical chain. In someembodiments, the second and sixth optical chains have the same focallength and capture the same portion of the scene of interest orsubstantially the same portion of the scene of interest.

Light redirection device position computation module 5408 is configuredto compute light redirection device positions based on the indicatedzoom setting prior to moving the first light redirection device inaccordance with the indicated zoom setting, e.g., for the first, second,third and fourth moveable light redirection devices of the first, third,fourth and fifth optical chains.

User zoom input module 5410 is configured to receive user inputindicating a zoom setting, e.g., from user input interface 5208.

FIG. 55 is a drawing of an exemplary stored zoom setting to lightredirection devices positions mapping table 5500 in accordance with anexemplary embodiment including four optical chains including moveablelight redirection devices. For example, table 5500 may be used whenimplementing a method in accordance with flowchart 5100 of FIG. 51; Themoveable light redirection devices are, e.g., mirrors, e.g.,substantially plane mirrors or prisms. In some embodiments, thesubstantially plane mirrors are plane mirrors. In this example, opticalchains 1, 3, 4, and 5 include moveable light redirection devices. Thefour moveable light redirection devices are, e.g., moveable controllablemirrors (5234, 5250, 5260, 5270) of camera 5200 of FIG. 52 or moveablecontrollable prisms (5334, 5350, 5360, 5370) of camera 5300 of FIG. 53.

First row 5502 includes 9 column headers (zoom setting, optical chain 1light redirection device 1 position, optical chain 1 light redirectiondevice 1 angle, optical chain 3 light redirection device 2 position,optical chain 3 light redirection device 2 angle, optical chain 4 lightredirection device 3 position, optical chain 4 light redirection device3 angle, optical chain 5 light redirection device 4 position, opticalchain 5 light redirection device 4 angle) which corresponds to columns(first column 5502, second column 5504, third column 5506, fourth column5508, fifth column 5510, sixth column 5512, seventh column 5514, eightcolumn 5516, ninth column 5518), respectively. In table 5500, the firstposition is the setting position for the light redirection device ofoptical chain1, which is moveable light redirection device 1; α is theangular setting corresponding to the first position; the second positionis the setting position for the light redirection device of opticalchain 3, which is moveable light redirection device 2, β is the angularsetting corresponding to the second position; the third position is thesetting position for the light redirection device of optical chain 4,which is moveable light redirection device 3; θ is the angular settingcorresponding to the third position; the fourth position is the settingposition for the light redirection device of optical chain 5, which ismoveable light redirection device 4, φ is the angular settingcorresponding to the fourth position;

Second row 5522 indicates that for the zoom setting of 1×, the firstposition=P1A; α=α1; the second position=P2A; β=β1; the thirdposition=P3A; θ=1; the fourth position=P4A; and φ=φ1. Third row 5524indicates that for the zoom setting of 2×, the first position=P1B; α=α2;the second position=P2B; β=β2; the third position=P3B; θ=θ2; the fourthposition=P4B; and φ=φ2. Fourth row 5526 indicates that for the zoomsetting of 3×, the first position=P1C; α=α3; the second position=P2C;β=β3; the third position=P3C; θ=θ3; the fourth position=P4C; and φ=φ3.Fifth row 5528 indicates that for the zoom setting of 4×, the firstposition=P1D; α=α4; the second position=P2D; β=β4; the thirdposition=P3D; θ=θ4; the fourth position=P4D; and φ=φ4.

FIG. 56 is a drawing 5600 illustrating an example in which a combinedimage is generated from captured images from multiple optical chains,corresponding to a zoom setting of 1×, in accordance with an exemplaryembodiment. For example, camera device 5200 of FIG. 52 or camera device5300 of FIG. 53 uses stored zoom light setting to light redirectiondevice position mapping table 5500 of FIG. 55 and implement a method inaccordance with flowchart 5100 of FIG. 51, to position moveable lightredirection devices in accordance with the zoom setting of 1×, capturingimages (5604, 5606, 5608, 5610, 5612, 5614, 5616) from optical chains,and generating combined image 5618. Area 5602 represents a scene area ofinterest. First image 5604 is of a first portion of the scene ofinterest 5602, and is captured using the first optical chain, with themoveable light redirection device of the first optical chain set so thatvariable first position=value P1A, and variable α=value α1, whichcorresponds to the 1× zoom setting for the first optical chain, aslisted in table 5500. Second image 5606 is of a second portion of thescene of interest 5602, and is captured using the second optical chain.Third image 5608 is of a third portion of the scene of interest 5602,and is captured using the third optical chain, with the moveable lightredirection device of the third optical chain set so that variablesecond position=value P2A, and variable β=value β1, which corresponds tothe 1× zoom setting for the third optical chain, as listed in table5500. Fourth image 5610 is of a fourth portion of the scene of interest5602, and is captured using the fourth optical chain, with the moveablelight redirection device of the fourth optical chain set so thatvariable third position=value P3A, and variable θ=value θ1, whichcorresponds to the 1× zoom setting for the fourth optical chain, aslisted in table 5500. Fifth image 5612 is of a fifth portion of thescene of interest 5602, and is captured using the fifth optical chain,with the moveable light redirection device of the fifth optical chainset so that variable fourth position=value P4A, and variable φ=value φ1,which corresponds to the 1× zoom setting for the fifth optical chain, aslisted in table 5500. Sixth image 5614 is of a sixth portion of thescene of interest 5602, and is captured using the sixth optical chain.Seventh image 5616 is of a seventh portion of the scene of interest5602, and is captured using the seventh optical chain.

The first image 5604 is of a first portion of the scene of interest 5602which corresponds to the upper left corner of the scene of interest. Thesecond image 5606 is of a second portion of the scene of interest 5602which corresponds to the entire area of the scene of interest orsubstantially the entire area of the scene of interest. The third image5608 is of a third portion of the scene of interest 5602 whichcorresponds to the lower left corner of the scene of interest. Thefourth image 5610 is of a fourth portion of the scene of interest 5602which corresponds to the upper right corner of the scene of interest.The fifth image 5612 is of a fifth portion of the scene of interest 5602which corresponds to the lower right corner of the scene of interest.The sixth image 5614 is of a sixth portion of the scene of interest 5602which corresponds to the entire area of the scene of interest orsubstantially the entire area of the scene of interest. The seventhimage 5616 is of a seventh portion of the scene of interest 5602 whichcorresponds to a center area of the scene of interest.

The combined image 5618, for the 1× setting, is generated by combiningthe captured images (5604, 5606, 5608, 5610, 5612, 5614, 5616) togenerate composite image 5618, which corresponds to the entire area ofthe scene or interest 5602 or substantially the entire area of the sceneof interest 5602. Note that the area of the center of the combined image5618 is generated using information from more captured images, than thearea near the outside of the combined image 5618.

FIG. 57 is a drawing 5700 illustrating an example in which a combinedimage is generated from captured images from multiple optical chains,corresponding to a zoom setting of 2×, in accordance with an exemplaryembodiment. For example, camera device 5200 of FIG. 52 or camera device5300 of FIG. 53 uses stored zoom light setting to light redirectiondevice position mapping table 5500 of FIG. 55 and implement a method inaccordance with flowchart 5100 of FIG. 51, to position moveable lightredirection devices in accordance with the zoom setting of 2×, capturingimages (5704, 5706, 5708, 5710, 5712, 5714, 5716) from optical chains,and generating combined image 5718. Area 5702 represents a scene area ofinterest. First image 5704 is of a first portion of the scene ofinterest 5702, and is captured using the first optical chain, with themoveable light redirection device of the first optical chain set so thatvariable first position=value P1B, and variable α=value α2, whichcorresponds to the 2× zoom setting for the first optical chain, aslisted in table 5500. Second image 5706 is of a second portion of thescene of interest 5702, and is captured using the second optical chain.Third image 5708 is of a third portion of the scene of interest 5702,and is captured using the third optical chain, with the moveable lightredirection device of the third optical chain set so that variablesecond position=value P2B, and variable β=value β2, which corresponds tothe 2× zoom setting for the third optical chain, as listed in table5500. Fourth image 5710 is of a fourth portion of the scene of interest5702, and is captured using the fourth optical chain, with the moveablelight redirection device of the fourth optical chain set so thatvariable third position=value P3B, and variable θ=value θ2, whichcorresponds to the 2× zoom setting for the fourth optical chain, aslisted in table 5500. Fifth image 5712 is of a fifth portion of thescene of interest 5702, and is captured using the fifth optical chain,with the moveable light redirection device of the fifth optical chainset so that variable fourth position=value P4B, and variable φ=value φ2,which corresponds to the 2× zoom setting for the fifth optical chain, aslisted in table 5500. Sixth image 5714 is of a sixth portion of thescene of interest 5702, and is captured using the sixth optical chain.Seventh image 5716 is of a seventh portion of the scene of interest5702, and is captured using the seventh optical chain.

The first image 5704 is of a first portion of the scene of interest 5702which corresponds to an area in the upper left area of the scene ofinterest, centered along a diagonal extending from the center of thescene of interest to the upper left corner of the scene of interest. Thesecond image 5706 is of a second portion of the scene of interest 5702which corresponds to the entire area of the scene of interest orsubstantially the entire area of the scene of interest. The third image5708 is of a third portion of the scene of interest 5702 whichcorresponds to the lower left area of the scene of interest, centeredalong a diagonal extending from the center of the scene of interest tothe lower left corner of the scene of interest. The fourth image 5710 isof a fourth portion of the scene of interest 5702 which corresponds tothe upper right area of the scene of interest, centered along a diagonalextending from the center of the scene of interest to the upper rightcorner of the scene of interest. The fifth image 5712 is of a fifthportion of the scene of interest 5702 which corresponds to the lowerright area of the scene of interest, centered along a diagonal extendingfrom the center of the scene of interest to the lower right corner ofthe scene of interest. The sixth image 5714 is of a sixth portion of thescene of interest 5702 which corresponds to the entire area of the sceneof interest or substantially the entire area of the scene of interest.The seventh image 5716 is of a seventh portion of the scene of interest5702 which corresponds to a center area of the scene of interest.

The combined image 5718, for the 2× zoom setting, is generated bycombining information from the captured images (5704, 5706, 5708, 5710,5712, 5714, 5716), e.g., overlapping areas within the captured images,which are within the area of composite image 5718 boundaries are used,to generate composite image 5718, which corresponds to an area which isone half or approximately one half the area of the scene of interest,and which is centered at the center of the scene of interest. Note thatcombined image 5718 is generated from first image 5704, cropped secondimage 5707, third image 5708, fourth image 5710, fifth image 5712,cropped sixth image 5715, and seventh image 5716. Note that the area ofthe center of the combined image 5718 is generated using informationfrom more captured images, than the area near the outside of thecombined image 5718.

FIG. 58 is a drawing 5800 illustrating an example in which a combinedimage is generated from captured images from multiple optical chains,corresponding to a zoom setting of 3×, in accordance with an exemplaryembodiment. For example, camera device 5200 of FIG. 52 or camera device5300 of FIG. 53 uses stored zoom light setting to light redirectiondevice position mapping table 5500 of FIG. 55 and implement a method inaccordance with flowchart 5100 of FIG. 51, to position moveable lightredirection devices in accordance with the zoom setting of 3×, capturingimages (5804, 5806, 5808, 5810, 5812, 5814, 5816) from optical chains,and generating combined image 5818. Area 5802 represents a scene area ofinterest. First image 5804 is of a first portion of the scene ofinterest 5802, and is captured using the first optical chain, with themoveable light redirection device of the first optical chain set so thatvariable first position=value P1C, and variable α=value α3, whichcorresponds to the 3× zoom setting for the first optical chain, aslisted in table 5500. Second image 5806 is of a second portion of thescene of interest 5802, and is captured using the second optical chain.Third image 5808 is of a third portion of the scene of interest 5802,and is captured using the third optical chain, with the moveable lightredirection device of the third optical chain set so that variablesecond position=value P2C, and variable β=value β3, which corresponds tothe 3× zoom setting for the third optical chain, as listed in table5500. Fourth image 5810 is of a fourth portion of the scene of interest5802, and is captured using the fourth optical chain, with the moveablelight redirection device of the fourth optical chain set so thatvariable third position=value P3C, and variable θ=value θ3, whichcorresponds to the 3× zoom setting for the fourth optical chain, aslisted in table 5500. Fifth image 5812 is of a fifth portion of thescene of interest 5802, and is captured using the fifth optical chain,with the moveable light redirection device of the fifth optical chainset so that variable fourth position=value P4C, and variable φ=value φ3,which corresponds to the 3× zoom setting for the fifth optical chain, aslisted in table 5500. Sixth image 5814 is of a sixth portion of thescene of interest 5802, and is captured using the sixth optical chain.Seventh image 5816 is of a seventh portion of the scene of interest5802, and is captured using the seventh optical chain.

The first image 5804 is of a first portion of the scene of interest 5802which corresponds to an area in the upper left area of the scene ofinterest, centered along a diagonal extending from the center of thescene of interest to the upper left corner of the scene of interest. Thesecond image 5806 is of a second portion of the scene of interest 5802which corresponds to the entire area of the scene of interest orsubstantially the entire area of the scene of interest. The third image5808 is of a third portion of the scene of interest 5802 whichcorresponds to the lower left area of the scene of interest, centeredalong a diagonal extending from the center of the scene of interest tothe lower left corner of the scene of interest. The fourth image 5810 isof a fourth portion of the scene of interest 5802 which corresponds tothe upper right area of the scene of interest, centered along a diagonalextending from the center of the scene of interest to the upper rightcorner of the scene of interest. The fifth image 5812 is of a fifthportion of the scene of interest 5802 which corresponds to the lowerright area of the scene of interest, centered along a diagonal extendingfrom the center of the scene of interest to the lower right corner ofthe scene of interest. The sixth image 5814 is of a sixth portion of thescene of interest 5802 which corresponds to the entire area of the sceneof interest or substantially the entire area of the scene of interest.The seventh image 5816 is of a seventh portion of the scene of interest5802 which corresponds to a center area of the scene of interest.

The combined image 5818, for the 3× zoom setting, is generated bycombining information from the captured images (5804, 5806, 5808, 5810,5812, 5814, 5816,), e.g., overlapping areas within the captured images,which are within the area of combined image 5718 boundaries are used, togenerate composite image 5818, which corresponds to an area which is onethird or approximately one third the area of the scene of interest, andwhich is centered at the center of the scene of interest. Note thatcombined image 5818 is generated from first image 5804, cropped secondimage 5807, third image 5808, fourth image 5810, fifth image 5812,cropped sixth image 5815, and seventh image 5816. Note that the area ofthe center of the combined image 5818 is generated using informationfrom more captured images, than the area near the outside of thecombined image 5818.

FIG. 59 is a drawing 5900 illustrating an example in which a combinedimage is generated from captured images from multiple optical chains,corresponding to a zoom setting of 4×, in accordance with an exemplaryembodiment. For example, camera device 5200 of FIG. 52 or camera device5300 of FIG. 53 uses stored zoom light setting to light redirectiondevice position mapping table 5500 of FIG. 55 and implement a method inaccordance with flowchart 5100 of FIG. 51, to position moveable lightredirection devices in accordance with the zoom setting of 4×, capturingimages (5904, 5906, 5908, 5910, 5912, 5914, 5916) from optical chains,and generating combined image 5918. Area 5902 represents a scene area ofinterest. First image 5904 is of a first portion of the scene ofinterest 5902, and is captured using the first optical chain, with themoveable light redirection device of the first optical chain set so thatvariable first position=value P1 D, and variable α=value α4, whichcorresponds to the 4× zoom setting for the first optical chain, aslisted in table 5500. Second image 5906 is of a second portion of thescene of interest 5902, and is captured using the second optical chain.Third image 5908 is of a third portion of the scene of interest 5902,and is captured using the third optical chain, with the moveable lightredirection device of the third optical chain set so that variablesecond position=value P2D, and variable β=value β4, which corresponds tothe 4× zoom setting for the third optical chain, as listed in table5500. Fourth image 5910 is of a fourth portion of the scene of interest5902, and is captured using the fourth optical chain, with the moveablelight redirection device of the fourth optical chain set so thatvariable third position=value P3D, and variable θ=value θ4, whichcorresponds to the 4× zoom setting for the fourth optical chain, aslisted in table 5500. Fifth image 5912 is of a fifth portion of thescene of interest 5902, and is captured using the fifth optical chain,with the moveable light redirection device of the fifth optical chainset so that variable fourth position=value P4D, and variable φ=value φ4,which corresponds to the 4× zoom setting for the fifth optical chain, aslisted in table 5500. Sixth image 5914 is of a sixth portion of thescene of interest 5902, and is captured using the sixth optical chain.Seventh image 5916 is of a seventh portion of the scene of interest5902, and is captured using the seventh optical chain.

The first image 5904 is of a first portion of the scene of interest 5902which corresponds to a center area of the scene of interest. The secondimage 5906 is of a second portion of the scene of interest 5902 whichcorresponds to the entire area of the scene of interest or substantiallythe entire area of the scene of interest. The third image 5908 is of athird portion of the scene of interest 5902 which corresponds a centerarea of the scene of interest. The fourth image 5910 is of a fourthportion of the scene of interest 5902 which corresponds to a center areaof the scene of interest. The fifth image 5912 is of a fifth portion ofthe scene of interest 5902 which corresponds to a center area of thescene of interest. The sixth image 5914 is of a sixth portion of thescene of interest 5902 which corresponds to the entire area of the sceneof interest or substantially the entire area of the scene of interest.The seventh image 5916 is of a seventh portion of the scene of interest5902 which corresponds to a center area of the scene of interest.

The combined image 5918, for the 4× zoom setting, is generated bycombining information from the captured images (5904, 5906, 5908, 5910,5912, 5914, 5916), e.g., overlapping areas within the captured images,to generate composite image 5918, which corresponds to an area which isone fourth or approximately one fourth the area of the scene ofinterest, and which is centered at the center of the scene of interest.Note that combined image 5918 is generated from first image 5904,cropped second image 5907, third image 5908, fourth image 5910, fifthimage 5912, cropped sixth image 5915, and seventh image 5916.

FIG. 60, comprising the combination of FIG. 60A, FIG. 60B, FIG. 60C andFIG. 60D, is a flowchart 6000 of an exemplary method of capturing imagesin accordance with an exemplary embodiment. The exemplary method offlowchart 6000 may be performed by a camera device including multipleoptical chains or by a camera device including multiple optical chainsand an image processing device. In some embodiments, the camera deviceis one of: a smart phone and a camera. In some embodiments, the imageprocessing device is a personal computer.

Operation of the exemplary method starts in step 6002 and proceeds tostep 6004. In step 6004 the camera device monitors for informationindicating a user control zoom in or user control zoom out. Step 6004 isbeing performed on an ongoing basis. Step 6004 may, and sometimes does,include step 6006 in which the camera device detects informationindicating that a user control zoom in or user control zoom outoperation is to be performed. Operation proceeds from step 6006 to step6008. If the detection is for user control zoom in, then operationproceeds from step 6008 to step 6010; however, if the detection is foruser control zoom out, then operation proceeds from step 6008, viaconnecting node 6012 to step 6014.

Returning to step 6010, in step 6010 the camera device implements a zoomin operation in response to user control. Step 6010 includes steps 6016and step 6018. In some embodiments, step 6010 further includes one orboth of steps 6020 and 6022. In step 6016 the camera device adjusts adirection of a first light redirection device of a first optical chainof the camera device, the first light redirection device being one of asubstantially plane mirror or a prism, said adjusting increasing anoverlap in scene areas captured by the first optical chain of saidcamera device and a second optical chain of said camera device. In someembodiments, the substantially plane mirror is a plane mirror.

In step 6018, the camera devices adjusts a direction of a second lightredirection device of the second optical chain of the camera device, thesecond light redirection device being one of a mirror or a prism, saidadjusting increasing an overlap in scene areas captured by the first andsecond optical chains. In step 6020, the camera devices adjusts adirection of a fourth light redirection device of a fourth optical chainof the camera device, the fourth light redirection device being one of amirror or a prism, said adjusting increasing an overlap in scene areascaptured by the first, second, and fourth optical chains. In step 6022,the camera device adjusts a direction of a fifth light redirectiondevice of a fifth optical chain of the camera device, the fifth lightredirection device being one of a mirror or a prism, said adjustingincreasing an overlap in scene areas captured by the first, second,fourth, and fifth optical chains.

Returning to step 6014, in step 6014 the camera device implements a zoomout operation in response to user control. Step 6014 includes steps 6026and step 6028. In some embodiments, step 6014 further includes one orboth of steps 6030 and 6032. In step 6026 the camera device adjusts adirection of the first light redirection device of the first opticalchain of the camera device, said adjusting decreasing an overlap inscene areas captured by the first optical chain of said camera deviceand a second optical chain of said camera device.

In step 6028, the camera device adjusts a direction of the second lightredirection device of the second optical chain of the camera device,said adjusting decreasing an overlap in scene areas captured by thefirst and second optical chains. In step 6030, the camera device adjustsa direction of the fourth light redirection device of the fourth opticalchain of the camera device, said adjusting decreasing an overlap inscene areas captured by the first, second, and fourth optical chains. Instep 6032, the camera device adjusts a direction of the fifth lightredirection device of the fifth optical chain of the camera device saidadjusting decreasing an overlap in scene areas captured by the first,second, fourth, and fifth optical chains.

Operation proceeds from step 6010 or step 6014, via connecting node B6024, to step 6036, 6038, and 6040. In some embodiments, operation alsoproceeds, via connecting node B 6024 to one or more or all of steps6042, 6044, 6046, 6048, 6050, 6052, and 6054.

In step 6036 the camera device captures an image of the scene area usingthe first optical chain. In step 6038 the camera device captures animage of the scene area using the second optical chain. In step 6040 thecamera device captures an image of the scene area using a third opticalchain. In step 6042 the camera device captures an image of the scenearea using the fourth optical chain. In step 6044 the camera devicecaptures an image of the scene area using the fifth optical chain. Instep 6046 the camera device captures an image of the scene area using asixth optical chain. In step 6048 the camera device captures an image ofthe scene area using a seventh optical chain. In step 6050 the cameradevice captures an image of the scene area using an eighth opticalchain. In step 6052 the camera device captures an image of the scenearea using a ninth optical chain. In step 6054 the camera devicecaptures an image of the scene area using a tenth optical chain.

Operation proceeds from steps 6036, 6038, 6040, 6042, 6044, 6046, 6048,6050, 6052 and 6054, via connecting node C 6056 to step 6058. In step6058 a composite image is generated in accordance with a zoom focallength setting. Step 6058 includes one or more or all of steps 6060,6062, 6064, 6066, 6068, 6070, and 6072. In some embodiments, step 6058includes one of steps 6060, 6062, 6064, 6066, 6068, 6070, and 6072.

In step 6060 a composite image is generated, in accordance with the zoomsetting from images captured by the first second and third opticalchains. In some such embodiments, step 6060 includes step 6074, in whicha crop operation is performed on the image captured by the third opticalchain when the zoom focal length setting is greater than f2.

In step 6062 a composite image is generated, in accordance with the zoomsetting from images captured by the first, second, third, fourth, andfifth optical chains. In some such embodiments, step 6062 includes step6076, in which a crop operation is performed on the image captured bythe third optical chain when the zoom focal length setting is greaterthan f2.

In step 6064 a composite image is generated, in accordance with the zoomsetting from images captured by the first, second, third, fourth, fifth,and sixth optical chains. In some such embodiments, step 6064 includesstep 6078, in which crop operations are performed on the images capturedby the third optical chain when the zoom focal length setting is greaterthan f2.

In step 6066 a composite image is generated, in accordance with the zoomsetting from images captured by the first, second, third, fourth, fifth,sixth, and seventh optical chains. In some such embodiments, step 6066includes step 6080, in which crop operations are performed on the imagescaptured by the third and seventh optical chains when the zoom focallength setting is greater than f2.

In step 6068 a composite image is generated, in accordance with the zoomsetting from images captured by the first, second, third, fourth, fifth,sixth, seventh, and eighth optical chains. In some such embodiments,step 6068 includes step 6082, in which crop operations are performed onthe images captured by the third, seventh and eighth optical chains whenthe zoom focal length setting is greater than f2.

In step 6070 a composite image is generated, in accordance with the zoomsetting from images captured by the first, second, third, fourth, fifth,sixth, seventh, eighth, and ninth optical chains. In some suchembodiments, step 6070 includes step 6084, in which crop operations areperformed on the images captured by the third, seventh, eighth and ninthoptical chains when the zoom focal length setting is greater than f2.

In step 6072 a composite image is generated, in accordance with the zoomsetting from images captured by the first, second, third, fourth, fifth,sixth, seventh, eighth, ninth, and tenth optical chains. In some suchembodiments, step 6072 includes step 6086, in which crop operations areperformed on the images captured by the third, seventh, eighth, ninthand tenth optical chains when the zoom focal length setting is greaterthan f2.

Operation proceeds from step 6058 to step 6088, in which the generatedcomposite image is stored in memory. In some embodiments, step 6058 and6088 are performed by the camera device which captured the images. Inother embodiments, steps 6058 and 6088 are performed by an imageprocessing device, e.g., a personal computer which received the capturedimages from the plurality of optical chains.

Operation proceeds from step 6088, via connecting node D 6090 to step6004.

In some embodiments, one or more or all of the optical chainscorresponding to focal length f2 do not need and do not include amirror, since f2 is a small focal length and the optical chain capturesa large area, and cropping is used to the extent necessary, based on theuser selected zoom setting.

In various embodiments, f1>f2. In some such embodiments, f1=2 f2. Invarious embodiments, maximum zoom in corresponds to user selected zoomfocal length setting=f1 with cropping for the f2 focal length chains.

In some embodiments, at maximum zoom in, overlap is total, with regardto the various chains used to generate the composite image. In someembodiments, at maximum zoom in, the moveable mirrors, e.g., of first,second fourth and fifth optical chains, are each set at a 45 degreeangle with respect the respective sensor of its optical chain.

In various embodiments, at maximum zoom out, the overlap is minimalbetween the chains with moveable light redirection devices, and themirror angles of the moveable mirrors of the individual optical chainswill depend on camera configuration.

In some embodiments, between maximum zoom in and maximum zoom out, thecropping of images captured by the f2 modules is implemented and theamount of cropping is a function of the user selected zoom setting. Invarious embodiments, there is maximum cropping for a user selected zoomsetting=f1, and there is no cropping for a user selected zoomsetting=f2.

FIG. 61 is a drawing of an exemplary camera device 6100 in accordancewith various exemplary embodiments. Camera device 6100 includes aprocessor 6102, e.g., a CPU, memory 6104, an assembly of modules 6106,e.g., an assembly of hardware modules such as circuits, a user inputinterface 6108, a display 6110, an I/O interface 6112, and a pluralityof optical chains (optical chain 1 6118, optical chain 2 6120, opticalchain 3 6122, optical chain 4 6124, optical chain 5 6126, optical chain6 6128, optical chain 7 6130, optical chain 8 6132, optical chain 96134, optical chain 10 6136) coupled together via a bus 6137 over whichthe various elements may interchange data and information. In someembodiments, optical chain 4 6124, optical chain 5 6126, optical chain 66128, optical chain 7 6130, optical chain 8 6132, optical chain 9 6134,and optical chain 10 6136 are optional. Display 6110, e.g., a liquidcrystal display (LCD), a light emitting diode (LED) display, an organiclight emitting diode (OLED) display or an active matrix organic lightemitting diode (AMOLED) display, is for displaying images, imageportions, user menus, etc., to the user of camera 6100. User inputinterface 6108 includes, e.g., switches, buttons, touch screeninterface, etc. for receiving input from the user. Received user inputincludes input indicating a user controlled zoom in or a user controlledzoom out. I/O interface 6112 includes a plurality of communicationsports for communicating with other devices, e.g., a personal computer.I/O interface 6112 in some embodiments, includes one or more wirelesscommunications interfaces. Information communicated over I/O interface6112 includes, e.g., combined images generated from captured images frommultiple optical chains, and/or captured images from individual opticalchains to be processed and/or combined externally, e.g., in a personalcomputer.

Memory 6104 includes assembly of modules 6114 and data/information 6116.Data/information 6116, in some embodiments, includes stored informationused to determine a light redirection device position of an opticalchain including moveable light redirection as a function of user zoom inand or user zoom out information, e.g. scale factors and/or calibrationinformation, for multiple optical chains with moveable light redirectiondevices. Data information 6116, in some embodiments, further includesstored information used to determine an amount of cropping to beperformed on an image captured from an optical chain corresponding tozoom setting, e.g., for one or more optical chains in which lightredirection devices are not moved as a function of zoom setting.

Optical chain 1 6118 includes a lens 6138, e.g., an outer lens, amoveable substantially plane mirror 6140, and an image sensor 6142. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 6140 includes a hinge mount.Optical chain 1 6118 further includes a mirror movement module 6144,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 6146, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 6142 is used to capture an image. Light passesthrough lens 6138 before reaching mirror 6140. Mirror 6140, which is alight redirection device, directs the light which has passed throughlens 6138 onto the image sensor 6142.

Optical chain 2 6120 includes a lens 6148, e.g., an outer lens, amoveable substantially plane mirror 6150, and an image sensor 6152. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 6150 includes a hinge mount.Optical chain 2 6120 further includes a mirror movement module 6154,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 6156, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 6152 is used to capture a an image. Light passesthrough lens 6148 before reaching mirror 6150. Mirror 6150, which is alight redirection device, directs the light which has passed throughlens 6148 onto the image sensor 6152.

Optical chain 3 6122 includes a lens 6158, e.g., an outer lens, a fixedmirror 6160, e.g., a substantially plane fixed mirror, and an imagesensor 6162. In some embodiments, the substantially plane fixed mirroris a fixed plane mirror. Image sensor 6162 is used to capture a image.Light passes through lens 6122 before reaching mirror 6160. Mirror 6160,which is a light redirection device, directs the light which has passedthrough lens 6158 onto the image sensor 6162.

Optical chain 4 6124 includes a lens 6164, e.g., an outer lens, amoveable substantially plane mirror 6166, and an image sensor 6168. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 6166 includes a hinge mount.Optical chain 4 6124 further includes a mirror movement module 6170,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 6172, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 6168 is used to capture an image. Light passesthrough lens 6164 before reaching mirror 6166. Mirror 6166, which is alight redirection device, directs the light which has passed throughlens 6164 onto the image sensor 6168.

Optical chain 5 6126 includes a lens 6174, e.g., an outer lens, amoveable substantially plane mirror 6176, and an image sensor 6178. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 6176 includes a hinge mount.Optical chain 5 6126 further includes a mirror movement module 6180,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 6182, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 6178 is used to capture a an image. Light passesthrough lens 6174 before reaching mirror 6176. Mirror 6176, which is alight redirection device, directs the light which has passed throughlens 6174 onto the image sensor 6178.

Optical chain 6 6128 includes a lens 6184, e.g., an outer lens, a fixedmirror 6185, e.g., a substantially plane fixed mirror, and an imagesensor 6186. In some embodiments, the substantially plane fixed mirroris a fixed plane mirror. Image sensor 6186 is used to capture a image.Light passes through lens 6184 before reaching mirror 6185. Mirror 6185,which is a light redirection device, directs the light which has passedthrough lens 6184 onto the image sensor 6186.

Optical chain 7 6130 includes a lens 6187, e.g., an outer lens, a fixedmirror 6188, e.g., a substantially plane fixed mirror, and an imagesensor 6189. In some embodiments, the substantially plane fixed mirroris a fixed plane mirror. Image sensor 6189 is used to capture a image.Light passes through lens 6187 before reaching mirror 6188. Mirror 6188,which is a light redirection device, directs the light which has passedthrough lens 6187 onto the image sensor 6189.

Optical chain 8 6132 includes a lens 6190, e.g., an outer lens, a fixedmirror 6191, e.g., a substantially plane fixed mirror, and an imagesensor 6192. In some embodiments, the substantially plane fixed mirroris a fixed plane mirror. Image sensor 6192 is used to capture a image.Light passes through lens 6190 before reaching mirror 6191. Mirror 6191,which is a light redirection device, directs the light which has passedthrough lens 6190 onto the image sensor 6192.

Optical chain 9 6134 includes a lens 6193, e.g., an outer lens, a fixedmirror 6194, e.g., a substantially plane fixed mirror, and an imagesensor 6195. In some embodiments, the substantially plane fixed mirroris a fixed plane mirror. Image sensor 6195 is used to capture a image.Light passes through lens 6193 before reaching mirror 6194. Mirror 6194,which is a light redirection device, directs the light which has passedthrough lens 6193 onto the image sensor 6195.

Optical chain 10 6136 includes a lens 6196, e.g., an outer lens, a fixedmirror 6197, e.g., a substantially plane fixed mirror, and an imagesensor 6198. In some embodiments, the substantially plane fixed mirroris a fixed plane mirror. Image sensor 6198 is used to capture a image.Light passes through lens 6196 before reaching mirror 6197. Mirror 6197,which is a light redirection device, directs the light which has passedthrough lens 6196 onto the image sensor 6198.

In one exemplary embodiment optical chain 1 6118, optical chain 2 6120,optical chain 4 6124, optical chain 5 6126 and optical chain 6 6128 havea focal length=f1; optical chain 3 6122, optical chain 7 6130, opticalchain 8 6132, optical chain 9 6134 and optical chain 10 6136 have afocal length=f2; and f1>f2.

In various embodiments, camera device 6100 includes a user selectablecontinuous zoom capability between zoom focal length setting of f2 andzoom focal length setting f1.

In some embodiments, the images captured by the optical chains in whichthe mirror does not move as a function of zoom setting (optical chain 36122, optical chain 6 6128, optical chain 7 6130, optical chain 8 6132,optical chain 9 6134, and optical chain 10 6136) are centered atsubstantially the same position irrespective of the zoom setting. Invarious embodiments, an image captured by an optical chain in which themirror moves a function of zoom setting (optical chain 1 6118, opticalchain 2 6120, optical chain 4 6124, optical chain 5 6126), captures animage which is centered as a function of the zoom settings. For example,the image captured by optical chain 1 6118 is located along a firstdiagonal between the center of the zoom image and a corner of the sceneof interest, as a function of the zoom setting; the image captured byoptical chain 2 6120 is located along a second diagonal between thecenter of the zoom image and a corner of the scene of interest, as afunction of the zoom setting; the image captured by optical chain 4 6124is located along a third diagonal between the center of the zoom imageand a corner of the scene of interest, as a function of the zoomsetting; and the image captured by optical chain 5 6126 is located alonga second diagonal between the center of the zoom image and a corner ofthe scene of interest, as a function of the zoom setting.

In some embodiments, an optical chain with a fixed mirror (optical chain3 6122, optical chain 6 6128, optical chain 7 6130, optical chain 86132, optical chain 9 6134, optical chain 10 6136) may be replaced by anoptical chain with a moveable mirror, in which the mirror remains in afixed position irrespective of the zoom setting.

In various embodiments, a different type of light redirection device isused instead of a mirror, e.g., a prism is used in place of a mirror inan optical chain. For example, a moveable prism is used in optical chain1 6118 in place of moveable substantially plane mirror 6140. In someembodiments, different types of light redirection devices are used indifferent optical chains. For example, in one embodiment, a moveablesubstantially plane mirror is used in optical chain 1 6118, and amoveable prism is used in optical chain 2 6120.

In some embodiments, an optical chain corresponding to focal length f2does not include a light redirection device and the image sensor of theoptical chain is positioned to be able to capture light entering thelens of the optical chain without redirection via a light redirectiondevice.

In some embodiments, one or more or all of the optical chainscorresponding to focal length f2, e.g., optical chain 3 6122, opticalchain 7 6130, optical chain 8 6132, optical chain 9 6134, and/or opticalchain 10 6136, do not need and do not include a mirror, since f2 is asmall focal length and the optical chain captures a large area, andcropping is used to the extent necessary, based on the user selectedzoom setting.

In various embodiments, f1>f2. In some such embodiments, f1=2 f2. Invarious embodiments, maximum zoom in corresponds to user selected zoomfocal length setting=f1 with cropping for the f2 focal length chains.

In some embodiments, at maximum zoom in, overlap is total, with regardto the various chains used to generate the composite image. In someembodiments, at maximum zoom in, the moveable mirrors, e.g., mirrors(6140, 6150, 6166, 6176), are set at a 45 degree angle with respect therespective sensor (6142, 6152, 6168, 6178), of the chain (6118, 6120,6124, 6126), respectively.

In various embodiments, at maximum zoom out, the overlap is minimalbetween the chains with moveable light redirection devices, and themirror angles of the moveable mirrors of the individual optical chainswill depend on camera configuration.

In some embodiments, between maximum zoom in and maximum zoom out, thecropping of images captured by the f2 modules, e.g. modules (6122, 6130,6132, 6134, 6134, 6136) is implemented and the amount of cropping is afunction of the user selected zoom setting. In various embodiments,there is maximum cropping for a user selected zoom setting=f1, and thereis no cropping for a user selected zoom setting=f2.

In one exemplary embodiment, camera 6100 of FIG. 61 is camera 1250 ofFIG. 12B, and optical chains (optical chain 1 6118, optical chain 26120, optical chain 3 6122, optical chain 4 6124, optical chain 5 6126,optical chain 6 6128, optical chain 7 6130, optical chain 8 6132,optical chain 9 6134, optical chain 10 6136) of FIG. 61 are opticalchains (1210, 1212, 1224, 1202, 1220, 1216, 1208, 1214, 1204, 1218),respectively. In some embodiments, optical chains (1210, 1212, 1202,1220, 1216) have a focal length of f1, and optical chains (1224, 1208,1214, 1204, 1218) have a focal length of f2. In such embodiments, themirrors of chains (1224, 1216, 1208, 1214, 1204, 1218), remained at afixed position, irrespective of the zoom focal length setting, e.g.,within the range of f2 to f1, while the mirrors of chains (1210, 1212,1202, 1220), are controllably positioned as a function of the userselected focal length setting, e.g., within the range of f2 to f1.

FIG. 62 is a drawing of an assembly of modules 6200, which may beincluded in a camera device implemented in accordance with the presentinvention, e.g., camera 6100 of FIG. 61. Assembly of modules 6200 mayimplement steps of a method, e.g., steps of the method of flowchart 6000of FIG. 60. In some embodiments, assembly of modules 6200 is an assemblyof circuits, which may be coupled together. In one exemplary embodiment,assembly of modules 6200 is assembly of modules 6106 of camera 6100 ofFIG. 61. In some embodiments, the assembly of module 6200 is an assemblyof software modules. In one exemplary embodiment, assembly of modules6200 is assembly of modules 6114 of memory 6104 of camera 6100 of FIG.61.

FIG. 62 illustrates an assembly of modules 6200 which can, and in someembodiments is, used in the camera device 6100 illustrated in FIG. 61.The modules in the assembly of modules 6200 can, and in some embodimentsare, implemented fully in hardware within the processor 6102, e.g., asindividual circuits. The modules in the assembly of modules 6200 can,and in some embodiments are, implemented fully in hardware within theassembly of modules 6106, e.g., as individual circuits corresponding tothe different modules. In other embodiments some of the modules areimplemented, e.g., as circuits, within the processor 6102 with othermodules being implemented, e.g., as circuits within assembly of modules6106, external to and coupled to the processor. As should be appreciatedthe level of integration of modules on the processor and/or with somemodules being external to the processor may be one of design choice.

Alternatively, rather than being implemented as circuits, all or some ofthe modules in assembly of modules 6100 may be implemented in softwareand stored in the memory 6104 of the camera device 6100, with themodules controlling operation of camera 6100 to implement the functionscorresponding to the modules when the modules are executed by aprocessor, e.g., processor 6102. In some such embodiments, the assemblyof modules 6200 is included in the memory 6104 as assembly of modules6114. In still other embodiments, various modules in assembly of modules6200 are implemented as a combination of hardware and software, e.g.,with another circuit external to the processor providing input to theprocessor 6102 which then under software control operates to perform aportion of a module's function. While shown in the FIG. 61 embodiment asa single processor, e.g., computer, it should be appreciated that theprocessor 6102 may be implemented as one or more processors, e.g.,computers.

When implemented in software the modules include code, which whenexecuted by the processor 6102, configure the processor 6102 toimplement the function corresponding to the module. In embodiments wherethe assembly of modules 6200 is stored in the memory 6104, the memory6104 is a computer program product comprising a computer readable mediumcomprising code, e.g., individual code for each module, for causing atleast one computer, e.g., processor 6102, to implement the functions towhich the modules correspond.

Completely hardware based or completely software based modules may beused. However, it should be appreciated that any combination of softwareand hardware, e.g., circuit implemented modules may be used to implementthe functions. As should be appreciated, the modules illustrated in FIG.62 control and/or configure the camera device 6100 or elements thereinsuch as the processor 6102, to perform functions of the correspondingsteps illustrated in the method flowchart 6000 of FIG. 60. Thus theassembly of modules 6200 includes various modules that perform functionsof the corresponding steps of the method shown in FIG. 60.

Assembly of modules 6200 includes a zoom input monitoring module 6202, azoom in light redirection device positioning adjustment control module6204, a zoom out light redirection device positioning adjustment controlmodule 6206, an image capture control module 6208, a composite imagegeneration module 6210, a cropping module 6212, and a composite imagestorage module 6214.

Zoom input monitoring module 6202 is configured to monitor forinformation indicating that a user control user input zoom or usercontrol zoom output is to be performed. Zoom input monitoring module6202 detects information indicating that a user control zoom in or auser control zoom output operation is to be performed. Detectedinformation includes, e.g., information identifying which one of a zoomin or zoom out is to be performed and/or information indicating theselected value for the zoom setting, e.g., at which the camera is tocapture images.

Zoom in light redirection device positioning adjustment control module6204 is configured to implement a zoom in operation in response to usercontrol, said zoom in operation including adjusting a direction of afirst light redirection device of a first optical chain of a camera,said first light redirection device being one of a substantially planemirror or a prism, said adjusting increasing an overlap in scene areascaptured by the first optical chain of the camera and a second opticalchain of the camera. In some embodiments, the direction of the firstlight redirection device is a direction perpendicular to a face of thefirst light redirection device.

In various embodiments, zoom in light redirection device positioningadjustment control module 6204 is further configured to adjust adirection of the second light redirection device of the second opticalchain, said second light redirection device being one of a mirror or aprism, said adjusting increasing an overlap in scene areas captured bythe first and second optical chains, as part of implementing a zoom inoperation.

In some embodiments, zoom in light redirection device positioningadjustment control module 6204 is further configured to adjust adirection of a fourth light redirection device of a fourth optical chainand a fifth light redirection device of a fifth optical chain, saidadjusting increasing the overlap in scene areas captured by the first,second, fourth and fifth optical chains.

Zoom out light redirection device positioning adjustment control module6206 is configured to implement a zoom out operation in response to usercontrol, said zoom out operation including adjusting the direction ofthe first light redirection device and the direction of the second lightredirection device, said adjusting decreasing an overlap in scene areascaptured by the first and second optical chains. In some embodiments,zoom out light redirection device positioning adjustment control module6206 is further configured to adjust a direction of a fourth lightredirection device of a fourth optical chain and a fifth lightredirection device of a fifth optical chain, said adjusting decreasingthe overlap in scene areas captured by the first, second, fourth andfifth optical chains.

Image capture control module 6208 is configured to: control the firstoptical chain to capture an image of a scene area, and control thesecond optical chain to capture an image of a scene area. Image capturecontrol module 6208 is further configured to control a third opticalchain of the camera device to capture an image of the scene area.

In some embodiment, the first and second optical chains have the samefocal length, f1. In some such embodiments, the third optical chain ofthe camera has a focal length f2, where f2 is smaller than f1. In someembodiments, the third optical chain has an optical axis which isperpendicular to the face of the optical chain and which remains fixedirrespective of a zoom setting, the range extending between f2 and f1.

In various embodiments, the zoom in and zoom out operation correspond toa zoom focal length setting in the range of f2 to f1, said rangeincluding f2 and f1.

In some embodiments, image capture control module 6208 is furtherconfigured to: control a fourth optical chain to capture an image of ascene area, and control the fifth optical chain to capture an image of ascene area. In some such embodiments, image capture control module 6208is further configured to control a sixth optical chain of the cameradevice to capture an image of the scene area. In some such embodiments,the first, second, fourth, fifth, and sixth optical chains have the samefocal length, e.g., f1.

In some embodiments, image capture control module 6208 is furtherconfigured to: control a seventh optical chain to capture an image of ascene area. In some such embodiments, image capture control module 6208is further configured to: control an eight optical chain to capture animage of a scene area. In some such embodiments, image capture controlmodule 6208 is further configured to: control a ninth optical chain tocapture an image of a scene area. In some such embodiments, imagecapture control module 6208 is further configured to: control a tenthoptical chain to capture an image of a scene area. In some suchembodiments, the third, seventh, eighth, ninth, and tenth optical chainshave the same focal length, e.g., f2.

Composite image generation module 6210 is configured to generate acomposite image, in accordance with a zoom focal length setting, fromcaptured images. In some embodiments, composite image generation module6210 is configured to generate a composite image in accordance with thezoom focal length setting from images captured by the first, second, andthird optical chains. In some embodiments, composite image generationmodule 6210 is configured to generate a composite image in accordancewith the zoom focal length setting from images captured by the first,second, third, fourth, and fifth optical chains. In some embodiments,composite image generation module 6210 is configured to generate acomposite image in accordance with the zoom focal length setting fromimages captured by the first, second, third, fourth, fifth, and sixthoptical chains. In some embodiments, composite image generation module6210 is configured to generate a composite image in accordance with thezoom focal length setting from images captured by the first, second,third, fourth, fifth, sixth, and seventh optical chains. In someembodiments, composite image generation module 6210 is configured togenerate a composite image in accordance with the zoom focal lengthsetting from images captured by the first, second, third, fourth, fifth,sixth, seventh, and eighth optical chains. In some embodiments,composite image generation module 6210 is configured to generate acomposite image in accordance with the zoom focal length setting fromimages captured by the first, second, third, fourth, fifth, sixth,seventh, eighth, and ninth optical chains. In some embodiments,composite image generation module 6210 is configured to generate acomposite image in accordance with the zoom focal length setting fromimages captured by the first, second, third, fourth, fifth, sixth,seventh, eighth, ninth, and tenth optical chains.

Cropping module 6212 is configured to perform a crop operation on theimage captured by the third optical chain when the zoom setting isgreater than f2. In various embodiments, the amount of crop is afunction of the difference between the zoom focal length setting and f2,and cropping module 6212 determines, e.g., calculates, the amount ofcrop prior to performing the crop.

In some embodiments, cropping module 6216 is configured to perform acrop operation on the image captured by the third and seventh opticalchain when the zoom setting is greater than f2. In some embodiments,cropping module 6216 is configured to perform a crop operation on theimage captured by the third, seventh, and eighth optical chains when thezoom setting is greater than f2. In some embodiments, cropping module6216 is configured to perform a crop operation on the image captured bythe third, seventh, eighth, and ninth optical chains when the zoomsetting is greater than f2. In some embodiments, cropping module 6216 isconfigured to perform a crop operation on the image captured by thethird, seventh, eighth, ninth, and tenth optical chains when the zoomsetting is greater than f2.

Composite image storage module 6214 is configured to store a generatedcomposite image in memory.

FIG. 63 is a drawing 6300 which illustrates exemplary composite images,corresponding to different zoom focal length settings, in accordancewith an exemplary embodiment. Drawing 6301 includes an exemplary sceneof interest 6302. Consider that an exemplary camera device, e.g., camera6100 of FIG. 62 and/or a camera implementing the method of flowchart6000 of FIG. 60 includes continuous zoom in/zoom out capability in arange of zoom focal length setting of f2 to f1, where f2<f1. Drawing6303 illustrates an exemplary composite image 6304 corresponding to auser selected zoom focal length setting of f2. In this example, thecomposite image 6304 corresponds to that full area of the scene ofinterest 6302. Drawing 6305 illustrates an exemplary composite image6306 corresponding to a user selected zoom focal length setting of f1.For this example, composite image 6306 corresponds to a maximum zoom in,and composite image 6304 corresponds to a maximum zoom out.

Drawing 6307 illustrates composite images (6304, 6310, 6312, 6314, 6316,6318, 6320, 6322, 6324, 6326, 6306) corresponding to a plurality ofexemplary different user selected zoom focal length settings. Note thatin this example, the zoom is about the center of the image. Going indirection6306->6310->6312->6314->6316->6318->6320->6322->6324->6326->6306 is thezoom in direction. Going in direction6306->6326->6324->6322->6320->6318->6316->6314->6312->6310->6304 is thezoom in direction. In various embodiments continuous zoom is possible inthe range of zoom focal length setting f2 to f1.

Drawing 6400 of FIG. 64 illustrates captured images which are combinedto generate a composite image for an example in which the user selectedfocal length setting is set at a maximum zoom out, e.g., zoom focallength setting=f2, in accordance with an exemplary embodiment. Considerthat an exemplary camera device, e.g., camera device 6100 of FIG. 62,and/or a camera device implementing the method of flowchart 6000 of FIG.60 includes continuous zoom in/zoom out capability in a range of zoomfocal length setting of f2 to f1, where f2<f1.

Drawing 6401 illustrates that optical chain 1 capture image 6404corresponds to a left upper portion of the scene of interest 6402.Drawing 6403 illustrates that optical chain 2 capture image 6406corresponds to a left lower portion of the scene of interest 6402.Drawing 6405 illustrates that optical chain 3 capture image 6408corresponds to the entire scene of interest 6402. Drawing 6407illustrates that optical chain 4 capture image 6410 corresponds to aright upper portion of the scene of interest 6402. Drawing 6409illustrates that optical chain 5 capture image 6412 corresponds to aright lower portion of the scene of interest 6402. Drawing 6411illustrates that optical chain 6 capture image 6414 corresponds to acenter portion of the scene of interest 6402. Drawing 6413 illustratesthat optical chain 7 capture image 6416 corresponds to the entire sceneof interest 6402. Drawing 6415 illustrates that optical chain 8 captureimage 6418 corresponds to the entire scene of interest 6402. Drawing6417 illustrates that optical chain 9 capture image 6420 corresponds tothe entire scene of interest 6402. Drawing 6419 illustrates that opticalchain 10 capture image 6422 corresponds to the entire scene of interest6402.

A composite image is generated, in accordance with the zoom focal lengthsetting, from captured images (6404, 6406, 6408, 6410, 6412, 6414, 6418,6420, 6422). The generated composite image is, e.g., composite image6304 of FIG. 63.

Drawing 6500 of FIG. 65 illustrates captured images and cropped capturedimages, some of which are combined to generate a composite image for anexample in which the user selected focal length setting is set at amaximum zoom in, e.g., zoom focal length setting=f1, in accordance withan exemplary embodiment. Consider that an exemplary camera device, e.g.,camera 6100 of FIG. 62, and/or a camera implementing the method offlowchart 6000 of FIG. 60 includes continuous zoom in/zoom outcapability in a range of zoom focal length setting of f2 to f1, wheref2<f1.

Drawing 6501 illustrates that optical chain 1 capture image 6504corresponds to a center portion of the scene of interest 6502. Drawing6503 illustrates that optical chain 2 capture image 6506 corresponds toa center portion of the scene of interest 6502. Drawing 6505 illustratesthat optical chain 3 capture image 6508 corresponds to the entire sceneof interest 6502. Drawing 6507 illustrates that optical chain 4 captureimage 6510 corresponds to a center portion of the scene of interest6502. Drawing 6509 illustrates that optical chain 5 capture image 6512corresponds to a center portion of the scene of interest 6502. Drawing6511 illustrates that optical chain 6 capture image 6514 corresponds toa center portion of the scene of interest 6502. Drawing 6513 illustratesthat optical chain 7 capture image 6516 corresponds to the entire sceneof interest 6502. Drawing 6515 illustrates that optical chain 8 captureimage 6518 corresponds to the entire scene of interest 6502. Drawing6517 illustrates that optical chain 9 capture image 6520 corresponds tothe entire scene of interest 6502. Drawing 6519 illustrates that opticalchain 10 capture image 6522 corresponds to the entire scene of interest6502.

In this example, since the user selected zoom focal length setting isf1, which is greater than f2, cropping operations are performed on theoptical chains with focal length=f2. The amount of the cropping is afunction of the difference between the zoom focal length setting and f2.Thus, captured images (6508, 6516, 6518, 6520, 6522) are cropped.

Drawing 6521 illustrates that optical chain 3 cropped capture image6524, corresponds to a center portion of the scene of interest 6502.Drawing 6523 illustrates that optical chain 7 cropped capture image6526, corresponds to a center portion of the scene of interest 6502.Drawing 6525 illustrates that optical chain 8 cropped capture image6528, corresponds to a center portion of the scene of interest 6502.Drawing 6527 illustrates that optical chain 9 cropped capture image6530, corresponds to a center portion of the scene of interest 6502.Drawing 6529 illustrates that optical chain 10 cropped capture image6532, corresponds to a center portion of the scene of interest 6502.

A composite image is generated, in accordance with the zoom focal lengthsetting, from optical chain 1 capture image 6504, optical chain 2capture image 6506, optical chain 3 cropped image 6524, optical chain 4capture image 6510, optical chain 5 capture image 6512, optical chain 6capture image 6514, optical chain 7 cropped image 6526, optical chain 8cropped image 6528, optical chain 9 cropped image 6530 and optical chain10 cropped image 6532. The generated composite image is, e.g., compositeimage 6306 of FIG. 63.

Drawing 6600 of FIG. 66 illustrates captured images and cropped capturedimages, some of which are combined to generate a composite image for anexample in which the user selected focal length setting is set at anintermediate value, e.g., a value between f2 and f1 in accordance withan exemplary embodiment. Consider that an exemplary camera device, e.g.,camera device 6100 of FIG. 62, and/or a camera implementing the methodof flowchart 6000 of FIG. 60 includes continuous zoom in/zoom outcapability in a range of zoom focal length setting of f2 to f1, wheref2<f1.

Drawing 6601 illustrates that optical chain 1 capture image 6604corresponds to a portion of the scene of interest 6602, which iscentered along a diagonal extending from the center of the scene ofinterest to the left upper corner of the scene of interest. Drawing 6603illustrates that optical chain 2 capture image 6606 corresponds to aportion of the scene of interest 6602, which is centered along adiagonal extending from the center of the scene of interest to a leftlower corner of the scene of interest. Drawing 6605 illustrates thatoptical chain 3 capture image 6608 corresponds to the entire scene ofinterest 6602. Drawing 6607 illustrates that optical chain 4 captureimage 6610 corresponds to a portion of the scene of interest 6602, whichis centered along a diagonal extending from the center of the scene ofinterest to the right upper corner of the scene of interest. Drawing6609 illustrates that optical chain 5 capture image 6612 corresponds toa portion of the scene of interest 6602, which is centered along adiagonal extending from the center of the scene of interest to the rightlower corner of the scene of interest. Drawing 6611 illustrates thatoptical chain 6 capture image 6614 corresponds to a center portion ofthe scene of interest 6602. Drawing 6613 illustrates that optical chain7 capture image 6616 corresponds to the entire scene of interest 6602.Drawing 6615 illustrates that optical chain 8 capture image 6618corresponds to the entire scene of interest 6602. Drawing 6617illustrates that optical chain 9 capture image 6620 corresponds to theentire scene of interest 6602. Drawing 6619 illustrates that opticalchain 10 capture image 6622 corresponds to the entire scene of interest6602.

In this example, since the user selected zoom focal length setting is avalue which is greater than f2, cropping operations are performed on theoptical chains with focal length=f2. The amount of the cropping is afunction of the difference between the zoom focal length setting and f2.Thus, captured images (6608, 6616, 6618, 6620, 6622) are cropped.

Drawing 6621 illustrates that optical chain 3 cropped capture image6624, corresponds to a center portion of the scene of interest 6602.Drawing 6623 illustrates that optical chain 7 cropped capture image6626, corresponds to a center portion of the scene of interest 6602.Drawing 6625 illustrates that optical chain 8 cropped capture image6628, corresponds to a center portion of the scene of interest 6602.Drawing 6627 illustrates that optical chain 9 cropped capture image6630, corresponds to a center portion of the scene of interest 6602.Drawing 6629 illustrates that optical chain 10 cropped capture image6632, corresponds to a center portion of the scene of interest 6602.

A composite image is generated, in accordance with the zoom focal lengthsetting, from optical chain 1 capture image 6604, optical chain 2capture image 6606, optical chain 3 cropped image 6624, optical chain 4capture image 6610, optical chain 5 capture image 6612, optical chain 6capture image 6614, optical chain 7 cropped image 6626, optical chain 8cropped image 6628, optical chain 9 cropped image 6630 and optical chain10 cropped image 6632.

The generated composite image is, e.g., composite image 6318 of FIG. 63.

FIG. 67 includes drawing 6650 which illustrates exemplary usercontrolled zoom and exemplary overlap between captured images fromdifferent optical chains, each including a moveable light redirectiondevice which is positioned, e.g., adjusted, as a function of a userselected zoom focal length setting, in accordance with an exemplaryembodiment.

Drawings 6651, 6653 and 6655 correspond to an example of a user selectedzoom focal length setting of f2, which corresponds to a maximum zoomout. Drawings 6667, 6669 and 6671 correspond to an example of a userselected zoom focal length setting of a value in between f2 and f1,where f1>f2, which corresponds to an intermediate zoom setting. Drawings6673, 6675 and 6677 correspond to an example of a user selected zoomfocal length setting f1, which corresponds to a maximum zoom in.

Legend 6668 identifies shading patterns used to illustrate the portionof the scene of interest captured by each of a plurality of opticalchains with a moveable light redirection device, e.g., a moveablesubstantially plane mirror or a prism. Vertical line shading asindicated by block 6670 is used to identify the image captured by afirst optical chain with a moveable light redirection device, e.g.,optical chain 1 6118 of camera device 6100 of FIG. 61. Horizontal lineshading as indicated by block 6672 is used to identify the imagecaptured by a second optical chain with a moveable light redirectiondevice, e.g., optical chain 2 6120 of camera device 6100 of FIG. 61.Negative slope line shading as indicated by block 6674 is used toidentify the image captured by a third optical chain with a moveablelight redirection device, e.g., optical chain 4 6124 of camera device6100 of FIG. 61. Positive slope line shading as indicated by block 6676is used to identify the image captured by a fourth optical chain with amoveable light redirection device, e.g., optical chain 5 6125 of cameradevice 6100 of FIG. 61.

Drawing 6653 illustrates that for maximum zoom out, which corresponds tozoom focal length setting of f2, the images captured by the four opticalchains including a moveable light redirection device are non-overlappingor substantially non-overlapping. Drawing 6655 illustrates that thecomposite image 6662 for the zoom focal length setting=f2, which isgenerated from images including the captured images from the fouroptical chains including a moveable light redirection device, asillustrated in drawing 6653, occupies the full area or substantially thefull area of the scene of interest 6652.

Drawing 6669 illustrates that for an intermediate zoom setting, whichcorresponds to zoom focal length setting value between f2 and f1, theimages captured by the four optical chains including a moveable lightredirection device are partially overlapping. As may be observed, themaximum overlap is in the center portion of the scene of interest.Drawing 6671 illustrates that the composite image 6664 for theintermediate zoom focal length setting, which is generated from imagesincluding the captured images from the four optical chains including amoveable light redirection device, as illustrated in drawing 6669,occupies a center portion of the scene of interest 6652.

Drawing 6675 illustrates that for a maximum zoom in, which correspondsto zoom focal length setting value of f1, the images captured by thefour optical chains including a moveable light redirection device arefully overlapping or substantially fully overlapping. Drawing 6677illustrates that the composite image 6666 for the maximum zoom in whichcorresponds to zoom focal length setting of f1, which is generated fromimages including the captured images from the four optical chainsincluding a moveable light redirection device, as illustrated in drawing6669, occupies the center portion of the scene of interest 6652. Notethat the size of the composite image 6666 is the same as orsubstantially the same as the size captured by an individual opticalchain with a moveable light redirection device.

Proceeding from drawing 6653 to drawing 6669 illustrates exemplary zoomin, and proceeding from drawing 6669 to drawing 6675 illustrates furtherzoom in. Proceeding from drawing 6675 to drawing 6669 illustratesexemplary zoom out, and proceeding from drawing 6669 to drawing 6653illustrates further zoom out.

It should be appreciated that captured images from other optical chains,may be, and in some embodiments, are further used in generating thecomposite image, e.g., captured images from optical chains in which alight redirection device is not moved as a function of the zoom focallength setting are further used in generating a composite image. In somesuch embodiments, cropping is used on some images captured by opticalchains in which a moveable light redirection device is not moved as afunction of zoom focal length setting, e.g., cropping is used on imagescaptured by optical chain 3 6122, optical chain 7 6128, optical chain 86132, optical chain 9 6134, optical chain 10 6136 of camera 6100 of FIG.61, with the amount of cropping being a function of the amount that theuser selected zoom focal length setting is greater than f2. For example,no cropping used in generating composite image 6652 and maximum croppingused in generating composite image 6666.

FIG. 68, comprising the combination of FIG. 68A and FIG. 68B, is aflowchart 6900 of an exemplary method of capturing images using a cameradevice including at least a first optical chain including a first sensorand a second optical chain including a second sensor, said first opticalchain having a smaller focal length than the second optical chain. Inone exemplary embodiment, the focal length of the first optical chain isf2 and the focal length of the second optical chain is f1, where f2<f1.

Operation of the exemplary method starts in step 6902, in which thecamera device is powered on and initialized, and proceeds to one or bothof optional steps 6904 and 6906.

In step 6904, the camera device receives user input. In someembodiments, step 6904 includes one or both of steps 6908 and 6910. Instep 6908 the camera device receives user input indicating a high rateof motion operation. In step 6910 the camera device receives user inputindicating a high resolution mode of operation.

Returning to step 6906, in step 6906 the camera device determines a rateof motion. In various embodiments, the step 6906 includes step 6911 inwhich the camera device determines the rate of motion from imagessequentially captured by an optical chain of said camera device. In somesuch embodiments, step 6911 includes step 6912 in which the cameradevice determines the rate of motion based on differences between animage captured during a first time period and an image captured during asecond time period which follows the first time period.

Operation proceeds from step 6904 and/or step 6906 to step 6914. In step6914, the camera device makes, e.g., in a control device of said cameradevice, a decision based on at least one of: i) user input or ii) adetermined rate of motion whether to control the second sensor of thesecond optical chain to perform joint readouts of pixel elementscorresponding to consecutive pixel element rows of the second sensor orto perform individual readouts of consecutive rows of pixel elements ofthe second sensor. In some embodiments, step 6914 includes one or bothof steps 6916 and 6918. In step 6916, the camera device makes thedecision based on user input. In step 6918, the camera device makes thedecision based on the determined rate of motion.

In various embodiments, step 6916 includes one or both of steps 6920 and6922. In step 6920 the camera device decides to perform joint readoutsof pixel elements corresponding to consecutive pixel element rows of thesecond sensor when the user input indicates selection of a high rate ofmotion mode of operation. In step 6922 the camera device decides toperform individual readouts of consecutive rows of pixel elements ofsaid second sensor when the user input indicates selection of a highresolution mode of operation.

In some embodiments, step 6918 includes one or both of steps 6924 and6926. In step 6924 the camera device decides to perform joint readoutsof pixel elements corresponding to consecutive pixel element rows of thesecond sensor when the rate of motion is above a first threshold. Instep 6926 the camera decides to perform individual readouts ofconsecutive rows of pixel elements of said second sensor when the rateof motion is below said first threshold.

Operation proceeds from step 6914, via connecting node A 6928, to steps6930 and step 6932. In some embodiments, in which the camera includes athird optical chain, operation also proceeds from step 6914, viaconnecting node A 6928 to step 6944. In step 6930 the camera controlsthe first sensor to readout pixel values of the first sensor. In someembodiments, step 6930 includes step 6936, in which the camera controlsthe first sensor to readout pixel values of pixel elements of the firstsensor at a rate which is a function of the ratio of the second focallength to the first focal length when said decision is a decision toperform individual readouts of consecutive rows of pixel elements ofsaid second sensor. In some such embodiments, step 6936 includes step6938 in which the camera device controls the first sensor to read outrows of pixel values at rate which is lower than the rate at which rowsof pixel values are read out of the second sensor.

Returning to step 6932, in step 6932 the camera device controls thesecond sensor to read out pixel values of pixel elements if the secondsensor in accordance with the decision to perform joint readouts ofpixel element rows of the second sensor or to perform individualreadouts of consecutive rows of pixel elements of said second sensor. Insome embodiments, step 6932 includes one or both of step 6940 and step6942. In step 6940 the camera device controls the second sensor to readout pixel values in parallel with the readout of at least some pixelvalues of the first sensor. In step 6942 the camera device controls thenumber of rows of pixel elements to be read out in a joint readout as afunction of the ratio of the first focal length to the second focallength when said decision is a decision to perform joint readouts ofpixel elements corresponding to consecutive pixel element rows of thesecond sensor.

Returning to step 6944, in step 6944 the camera device controls a thirdsensor corresponding to a third optical chain having the same focallength as the second optical china to perform joint readouts of pixelelements corresponding to consecutive pixel element rows when saiddecision is a decision to perform joint readouts of pixel elementscorresponding to consecutive pixel elements of the second sensor, saidthird sensor being controlled to jointly readout at some of pixel valuesafter the readout of the second sensor has been completed but prior tocompletion of the readout of the first sensor. Operation proceeds fromsteps 6930, 6932 and 6944, via connecting node B 6946, to the input tosteps 6904 and 6906.

FIG. 69 is a drawing of an exemplary camera device 7000 in accordancewith various exemplary embodiments. Camera device 7000 includes aprocessor 7002, e.g., a CPU, memory 7004, an assembly of modules 7006,e.g., an assembly of hardware modules such as circuits, a user inputinterface 7008, a display 7010, an I/O interface 7012, and a pluralityof optical chains (optical chain 1 7018, optical chain 2 7020, opticalchain 3 7022, optical chain 4 7024, optical chain 5 7026 . . . , opticalchain N 7028) coupled together via a bus 7037 over which the variouselements may interchange data and information. In some embodiments,optical chain 3 7022, optical chain 4 7024, optical chain 5 7026, . . ., optical chain N 7028 are optional, and one or more of the opticalchains may be included. Display 7010, e.g., a liquid crystal display(LCD), a light emitting diode (LED) display, an organic light emittingdiode (OLED) display or an active matrix organic light emitting diode(AMOLED) display, is for displaying images, image portions, user menus,etc., to the user of camera 7000. User input interface 7008 includes,e.g., switches, buttons, touch screen interface, etc. for receivinginput from the user. Received user input includes, e.g., inputindicating that a user has selected a high rate of motion mode ofoperation or input indicating that the user has selected a highresolution mode of operation. In some embodiments, the various modes ofoperation can be, and sometimes are, selected from a menu of user modesof operation, e.g., a menu of alternative mode of operation. I/Ointerface 7012 includes a plurality of communications ports forcommunicating with other devices, e.g., a personal computer. I/Ointerface 7012 in some embodiments, includes one or more wirelesscommunications interfaces. Information communicated over I/O interface7012 includes, e.g., combined images generated from captured images frommultiple optical chains, and/or captured images from individual opticalchains to be processed and/or combined externally, e.g., in a personalcomputer.

Memory 7004 includes assembly of modules 7014 and data/information 7016.Data/information 7016, in some embodiments, includes one or more or allof: received user input indicating a selected mode of operation, astored rate of motion threshold used in making a decision as to whetherto perform joint readouts or individual readouts, e.g., with regard tothe second sensor, a determined rate of motion, the focal length valuefor the first optical chain, the focal length value for the secondoptical chain, readouts from image sensors of optical chains, adetermined number of rows to be read in a joint readout, readout ratetiming information corresponding to the first sensor, and readout ratetiming information corresponding to the second sensor.

Optical chain 1 7018 includes a lens 7038, e.g., an outer lens, a fixedmirror 7040, e.g., a substantially plane fixed mirror, and an imagesensor 7042. In some embodiments, the substantially plane fixed mirroris a fixed plane mirror. Image sensor 7042 is used to capture a image.Light passes through lens 7038 before reaching mirror 7040. Mirror 7040,which is a light redirection device, directs the light which has passedthrough lens 7038 onto the image sensor 7042.

Optical chain 2 7020 includes a lens 7044, e.g., an outer lens, amoveable substantially plane mirror 7046, and an image sensor 7048. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 7046 includes a hinge mount.Optical chain 2 7020 further includes a mirror movement module 7050,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 7052, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 7048 is used to capture a an image. Light passesthrough lens 7044 before reaching mirror 7046. Mirror 7046, which is alight redirection device, directs the light which has passed throughlens 7044 onto the image sensor 7048.

Optical chain 3 7022 includes a lens 7054, e.g., an outer lens, amoveable substantially plane mirror 7056, and an image sensor 7058. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 7056 includes a hinge mount.Optical chain 3 7022 further includes a mirror movement module 7060,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 7062, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 7058 is used to capture a an image. Light passesthrough lens 7054 before reaching mirror 7056. Mirror 7056, which is alight redirection device, directs the light which has passed throughlens 7054 onto the image sensor 7058.

Optical chain 4 7024 includes a lens 7064, e.g., an outer lens, amoveable substantially plane mirror 7066, and an image sensor 7068. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 7066 includes a hinge mount.Optical chain 4 7024 further includes a mirror movement module 7070,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 7072, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 7068 is used to capture a an image. Light passesthrough lens 7064 before reaching mirror 7066. Mirror 7066, which is alight redirection device, directs the light which has passed throughlens 7064 onto the image sensor 7068.

Optical chain 5 7026 includes a lens 7074, e.g., an outer lens, amoveable substantially plane mirror 7076, and an image sensor 7078. Insome embodiments, the substantially plane mirror is a plane mirror. Invarious embodiments, the moveable mirror 7076 includes a hinge mount.Optical chain 5 7026 further includes a mirror movement module 7080,e.g., a drive motor and corresponding drive circuitry, and a mirrorposition detection module 7082, e.g., an angular position sensing devicesuch as a resolver or synchro or optical rotary encoder or a magneticrotary encoder, and, in some embodiments, circuitry to convert thesensor measurement into a digital value, e.g., an analog to digitalconverter. Image sensor 7078 is used to capture a an image. Light passesthrough lens 7074 before reaching mirror 7076. Mirror 7076, which is alight redirection device, directs the light which has passed throughlens 7074 onto the image sensor 7078.

Additional optical chains, e.g. optical chain N 7028, may include afixed or a moveable light redirection device, e.g., mirror.

In one exemplary embodiment optical chain 1 7018 has a focal length=f2;and each of optical chain 2 7020, optical chain 3 7022, optical chain 46124, and optical chain 5 6126 have a focal length=f1; and f1>f2.

In various embodiments, a different type of light redirection device isused instead of a mirror, e.g., a prism is used in place of a mirror inan optical chain. For example, a moveable prism is used in optical chain2 7020 in place of moveable substantially plane mirror 7046. In someembodiments, different types of light redirection devices are used indifferent optical chains.

In some embodiments, an optical chain corresponding to focal length f2does not include a light redirection device and the image sensor of theoptical chain is positioned to be able to capture light entering thelens of the optical chain without redirection via a light redirectiondevice.

In various embodiments, f1>f2. In some such embodiments, f1=2 f2.

In one exemplary embodiment, camera 7000 of FIG. 69 is camera 1250 ofFIG. 12B, and optical chains (optical chain 1 7018, optical chain 27020, optical chain 3 7022, optical chain 4 7024, optical chain 5 7026)of FIG. 69 are optical chains (1202, 1210, 1212, 1224, 1220),respectively of camera 1250 of FIG. 12B.

FIG. 70 is a drawing of an assembly of modules 7100, which may beincluded in a camera device implemented in accordance with the presentinvention, e.g., camera device 7000 of FIG. 69. Assembly of modules 7100may implement steps of a method, e.g., steps of the method of flowchart6900 of FIG. 68. In some embodiments, assembly of modules 7100 is anassembly of circuits, which may be coupled together. In one exemplaryembodiment, assembly of modules 7100 is assembly of modules 7006 ofcamera 7000 of FIG. 69. In some embodiments, the assembly of module 7100is an assembly of software modules. In one exemplary embodiment,assembly of modules 7100 is assembly of modules 7014 of memory 7004 ofcamera 7000 of FIG. 69.

FIG. 70 illustrates an assembly of modules 7100 which can, and in someembodiments is, used in the camera 7000 illustrated in FIG. 69. Themodules in the assembly of modules 7100 can, and in some embodimentsare, implemented fully in hardware within the processor 7002, e.g., asindividual circuits. The modules in the assembly of modules 7100 can,and in some embodiments are, implemented fully in hardware within theassembly of modules 7006, e.g., as individual circuits corresponding tothe different modules. In other embodiments some of the modules areimplemented, e.g., as circuits, within the processor 7002 with othermodules being implemented, e.g., as circuits within assembly of modules7006, external to and coupled to the processor. As should be appreciatedthe level of integration of modules on the processor and/or with somemodules being external to the processor may be one of design choice.

Alternatively, rather than being implemented as circuits, all or some ofthe modules in assembly of modules 7100 may be implemented in softwareand stored in the memory 7004 of the camera 7000, with the modulescontrolling operation of camera 7000 to implement the functionscorresponding to the modules when the modules are executed by aprocessor, e.g., processor 7002. In some such embodiments, the assemblyof modules 7100 is included in the memory 7004 as assembly of modules7014. In still other embodiments, various modules in assembly of modules7100 are implemented as a combination of hardware and software, e.g.,with another circuit external to the processor providing input to theprocessor 7002 which then under software control operates to perform aportion of a module's function. While shown in the FIG. 69 embodiment asa single processor, e.g., computer, it should be appreciated that theprocessor 7002 may be implemented as one or more processors, e.g.,computers.

When implemented in software the modules include code, which whenexecuted by the processor 7002, configure the processor 7002 toimplement the function corresponding to the module. In embodiments wherethe assembly of modules 7100 is stored in the memory 7004, the memory7004 is a computer program product comprising a computer readable mediumcomprising code, e.g., individual code for each module, for causing atleast one computer, e.g., processor 7002, to implement the functions towhich the modules correspond.

Completely hardware based or completely software based modules may beused. However, it should be appreciated that any combination of softwareand hardware, e.g., circuit implemented modules may be used to implementthe functions. As should be appreciated, the modules illustrated in FIG.71 control and/or configure the camera 7000 or elements therein such asthe processor 7002, to perform functions of the corresponding stepsillustrated in the method flowchart 6900 of FIG. 68. Thus the assemblyof modules 7100 includes various modules that perform functions of thecorresponding steps of the method shown in FIG. 68.

FIG. 70 is an assembly of modules 7100 in accordance with variousexemplary embodiments. Assembly of modules 7100 includes a readoutcontrol decision module 7102, a sensor readout control module 7104, amode control module 7106, a motion rate determination module 7108 and ajoint readout control module 7110.

In various embodiments, assembly of modules 7100 is included in a cameradevice, e.g. camera device 7000 of FIG. 69, including first opticalchain, e.g., optical chain 1 7018, including a first sensor, e.g.,sensor 7042, and having a first focal length, e.g., f2, and a secondoptical chain, e.g. optical chain 2 7020, including a second sensor,e.g., sensor 7048, and having a second focal length, e.g., f1, saidfirst focal length being smaller than said second focal length.

Readout control decision module 7102 is configured to make a readoutcontrol decision, based on at least one of: i) user input or ii) adetermined rate of motion, whether to control the second sensor of thesecond optical chain to perform joint readouts of pixel elementscorresponding to consecutive pixel element rows of the second sensor orto perform individual read outs of consecutive rows of pixel elements ofthe second sensor.

Sensor readout control module 7104 is configured to: i) control thefirst sensor to readout pixel values of said first sensor; and ii)control the second sensor to readout pixel values of pixel elements ofthe second sensor in accordance with the readout control decision.

In various embodiments, the sensor readout control module 7104 isconfigured to control said second sensor to perform joint readouts ofpixel elements corresponding to consecutive pixel element rows of thesecond sensor when said sensor control decision is a decision to controlthe second sensor to perform joint readouts of pixel elementscorresponding to consecutive pixel element rows of the second sensor andto control the second sensor to perform individual read outs ofconsecutive rows of pixel elements of said second sensor when saidsensor control decision is a decision to perform individual read outs ofconsecutive rows of pixel elements of said second sensor. In some suchembodiments, said sensor readout control module 7104 controls the firstand second sensors so that they readout rows of pixel valuescorresponding to the same portions of an image area of interest in asynchronized manner. In some embodiments, the first and second sensorscorrespond to overlapping image portions, and said sensor readoutcontrol module 7104 is configured to control the first and secondsensors to readout pixel values corresponding to the same portion of theimage area of interest in parallel.

In some embodiments, said sensor readout control module 7104 controlsthe second sensor to readout pixel values in parallel with the readoutof at least some pixel values of said first sensor, as part ofcontrolling the second sensor to readout pixel values.

In some embodiments, a user input interface, e.g., user input interface7008 of camera 7000 of FIG. 69, is configured to receive user inputindicating an automatic mode of operation, a high rate motion mode ofoperation or a high resolution mode of operation. Mode control module7106 is configured to set the camera device in a mode of operation inresponse to received user input. In various embodiments, readout controldecision module 7102 is configured to make said decision based on a rateof motion detected in images captured by said camera device whenreceived user input indicates an automatic mode of operation.

Readout control decision module 7102, in some embodiments, is configuredto: i) decide to perform joint readouts of pixel elements correspondingto consecutive pixel element rows of the second sensor when the rate ofmotion is above a first threshold; and ii) decide to perform individualread outs of consecutive rows of pixel elements of said second sensorwhen the rate of motion is below said first threshold.

Motion rate determination module 7108 is configured to determine saidrate of motion from images sequentially captured by an optical chain ofsaid camera device, e.g., prior to the readout control decision module7102 making a decision. In various embodiments, motion ratedetermination module 7108 determines the rate of motion based ondifferences between an image captured during a first time period and animage captured during a second time period which follows said first timeperiod.

In some embodiments, readout control decision module 7102 is configuredto: i) decide to perform joint readouts of pixel elements correspondingto consecutive pixel element rows of the second sensor when the userinput indicates selection of a high rate of motion mode of operation;and ii) decide to perform individual read outs of consecutive rows ofpixel elements of said second sensor when the user input indicatesselection of a high resolution motion mode of operation.

In various embodiments, the sensor readout control module 7104 isfurther configured to: control a third sensor, e.g., sensor 7058,corresponding to a third optical chain, e.g. optical chain 3 7022,having the same focal length as said second optical chain to performjoint readouts of pixel elements corresponding to consecutive pixelelement rows when said decision is a decision to perform joint readoutsof pixel elements corresponding to consecutive pixel element rows of thesecond sensor, said third sensor being controlled to jointly read out atleast some rows of pixel values after the readout of said second sensorhas been completed but prior to completion of the readout of the firstsensor.

Joint readout control module 7110 is configured to control the number ofrows of pixel elements to be read out in a joint read as a function ofthe ratio of the first focal length to the second focal length when saiddecision is a decision to perform joint readouts of pixel elementscorresponding to consecutive pixel element rows of the second sensor.

In some embodiments, the sensor readout control module 7104 is furtherconfigured to: control the first sensor to readout pixel values of pixelelements of said first sensor at a rate which is a function of the ratioof the second focal length to the first focal length when said decisionis a decision to perform individual read outs of consecutive rows ofpixel elements of said second sensor. In some such embodiments, saidsensor readout control module 7104 is further configured to: control thefirst sensor to read out rows of pixel values at a rate which is lowerthan the rate at which rows of pixel values are read out of the secondsensor, as part of being configured to control the first sensor toreadout pixel values of pixel elements of said first sensor at a ratewhich is based is a function of the ratio of the second focal length tothe first focal length.

FIG. 71 is a drawing 7200 illustrating an exemplary scene of interest7202 and exemplary capture areas corresponding to different opticalchains in a camera device in accordance with an exemplary embodiment.Area 7204 represents a sensor capture area for the sensor of opticalchain 1 with a focal length=f2. Area 7206 represents a sensor capturearea for the sensor of optical chain 2 with a focal length=f1. Area 7028represents a sensor capture area for the sensor of optical chain 3 witha focal length=f1. In one example, the camera device is camera device7000 of FIG. 69, which implements a method in accordance with flowchart6900 of FIG. 68 and includes the assembly of modules 7100 of FIG. 70.

FIG. 72 is a drawing 7300 illustrating exemplary readout times for anexample corresponding to user input high rate of motion mode ofoperation or automatic mode of operation with a high sensed rate. Firstsensor 7302, which is included in optical chain 1 corresponds to capturearea 7204 of FIG. 71. Second sensor 7304, which is included in opticalchain 2 corresponds to capture area 7206 of FIG. 71. Third sensor 7306,which is included in optical chain 3 corresponds to capture area 7208 ofFIG. 71. Column 7308 lists row readout times corresponding to each ofthe pixel element rows of first sensor 7302. Column 7310 lists joint rowreadout times corresponding to each of the pairs of consecutive pixelelement rows of second sensor 7304 and third sensor 7306.

In this example, in order to achieve a fast read, e.g., to avoid orreduce blurring, joint readouts of pixel elements corresponding toconsecutive pixel element rows of second and third sensors areperformed. During time 0->TR, the first row of the first sensor is readout and a joint readout is performed for the first and second rows ofthe second sensor. During time TR->2TR, the second row of the firstsensor is read out and a joint readout is performed for the third andfourth rows of the second sensor. During time 2TR->3TR, the third row ofthe first sensor is read out and a joint readout is performed for thefifth and sixth rows of the second sensor. During time 3TR->4TR, thefourth row of the first sensor is read out and a joint readout isperformed for the seventh and eight rows of the second sensor. Duringtime 4TR->5TR, the fifth row of the first sensor is read out and a jointreadout is performed for the ninth and tenth rows of the second sensor.During time 5TR->6TR, the sixth row of the first sensor is read out anda joint readout is performed for the first and second rows of the thirdsensor. During time 6TR->7TR, the seventh row of the first sensor isread out and a joint readout is performed for the third and fourth rowsof the third sensor. During time 7TR->8TR, the eighth row of the firstsensor is read out and a joint readout is performed for the fifth andsixth rows of the third sensor. During time 8TR->9TR, the ninth row ofthe first sensor is read out and a joint readout is performed for theseventh and eight rows of the third sensor. During time 9TR->10TR, thetenth row of the first sensor is read out and a joint readout isperformed for the ninth and tenth rows of the third sensor.

FIG. 73 is a drawing 7400 illustrating exemplary readout times for anexample corresponding to user input high resolution mode of operation orautomatic mode of operation with a low sensed rate. First sensor 7402,which is included in optical chain 1 corresponds to capture area 7204 ofFIG. 71. Second sensor 7404, which is included in optical chain 2corresponds to capture area 7206 of FIG. 71. Third sensor 7406, which isincluded in optical chain 3 corresponds to capture area 7208 of FIG. 71.Column 7408 lists row readout times corresponding to each of the pixelelement rows of first sensor 7402. Column 7410 lists individual rowreadout times corresponding to each of the pairs of consecutive pixelelement rows of second sensor 7404 and third sensor 7406.

In this example, in order to achieve high resolution, individualreadouts of pixel elements corresponding to pixel element rows of secondand third sensors are performed. During time 0->2TR, the first row ofthe first sensor is read out; during the time 0->TR the first row of thesecond sensor is read out; and during the time 1TR->2TR the second rowof the second sensor is read out. During time 2TR->4TR, the second rowof the first sensor is read out; during the time 2TR->3TR the third rowof the second sensor is read out; and during the time 3TR->4TR thefourth row of the second sensor is read out. During time 4TR->6TR, thethird row of the first sensor is read out; during the time 4TR->5TR thefifth row of the second sensor is read out; and during the time 5TR->6TRthe sixth row of the second sensor is read out. During time 6TR->8TR,the fourth row of the first sensor is read out; during the time 6TR->7TRthe seventh row of the second sensor is read out; and during the time7TR->8TR the eighth row of the second sensor is read out. During time8TR->10TR, the fifth row of the first sensor is read out; during thetime 8TR->9TR the ninth row of the second sensor is read out; and duringthe time 9TR->10TR the tenth row of the second sensor is read out.During time 10->12TR, the sixth row of the first sensor is read out;during the time 10->11TR the first row of the third sensor is read out;and during the time 11TR->12TR the second row of the third sensor isread out. During time 12TR->14TR, the seventh row of the first sensor isread out; during the time 12TR->13TR the third row of the third sensoris read out; and during the time 13TR->14TR the fourth row of the thirdsensor is read out. During time 14TR->16TR, the eighth row of the firstsensor is read out; during the time 14TR->15TR the fifth row of thethird sensor is read out; and during the time 15TR->16TR the sixth rowof the third sensor is read out. During time 16TR->18TR, the ninth rowof the first sensor is read out; during the time 16TR->17TR the seventhrow of the third sensor is read out; and during the time 17TR->18TR theeighth row of the third sensor is read out. During time 18TR->20TR, thetenth row of the first sensor is read out; during the time 18TR->19TRthe ninth row of the third sensor is read out; and during the time19TR->20TR the tenth row of the third sensor is read out.

Note that the overall readout time for the example of FIG. 73 is twiceas long as in the example of FIG. 72.

Various features are directed to methods and apparatus for reducing thethickness of a camera apparatus using one or more light redirectionelements are described. In various embodiments, the path of light alongthe optical axis is redirected prior to reaching a sensor. In at leastsome embodiments, the path of light entering the front of a cameradevice along the optical axis is redirected so it travels at leastpartially, in a direction which extends parallel to the face of thecamera. Accordingly, in at least some embodiments the length of the pathof the light is not limited by the depth, e.g., front to back length, ofthe camera.

The use of a light redirection element, such as, for example, a mirroror prism, is useful in extending the length of the light path from thefront most lens to the corresponding sensor, e.g., of an optical chain,e.g., camera module, in a camera which includes one or more cameramodules. In some embodiments, round lenses, e.g., lenses with roundapertures, are used as the outermost camera lenses and light redirectionelements which have a depth comparable to the diameter of the outer mostlens are used. In this manner the light received in what may beconsidered both the vertical and horizontal dimensions of the outer lensor lenses, assuming a camera is arranged vertically and the lens orlenses are mounted in a vertical plane corresponding to the front of acamera housing, can be redirected and captured without requiring thecamera to be sufficiently deep to allow the optical chain to be arrangedin a straight front to back configuration where the front outermost lensof the optical chain is positioned at the front of the camera and thesensor is positioned at the back of the camera directly behind theoutermost lens of the optical chain.

In some embodiments, a camera is implemented using multiple opticalchains, e.g., camera modules. In at least some such embodiments,redirection of light is implemented so that at least a portion of acamera module can take advantage of the left to right dimension, e.g.,side to side dimension of a camera, or up to down dimension (height)thereby allowing a relatively long focal length to be supported in arelatively thin camera format. In one such embodiment camera moduleswith small focal lengths and corresponding small lenses are implementedin a straight front to back implementation while camera modules withlarger focal length and larger lenses take advantage of lightredirection, e.g., by 90 degrees in some embodiments, to allow a portionof the camera module with the higher focal length to be implemented inthe sideways or up-down direction relative to the front of the camera.

While round lenses are used for camera modules in many embodiments, inat least some embodiments camera modules with long focal lengths areimplemented using lenses with non-round, e.g., oval, elongated or otherlens configurations which have apertures which are larger in onedimension than the other. In at least one such embodiment, the non-roundlenses are used in combination with light redirection devices. Asexplained in the detailed description the use of non-round lenses incombination with one or more light redirection devices can be used toimplement a camera module which is thinner than might be possible if alens with a round aperture was used instead.

In at least some embodiments, a combination of camera modules some withround lenses and others with non-round lenses are used. The non-roundlenses, e.g., lenses with apertures longer in one dimension of a planethan in another dimension of the same plane, are used for camera moduleshaving a large focal length, e.g., a focal length equivalent to a 70 mmfocal length of a full frame DSLR or greater in some embodiments,equivalent to a 150 mm focal length of a full frame DSLR or greater insome embodiments, equivalent to a 300 mm focal length of a full frameDSLR or greater in some embodiments. In at least some such embodiments,lenses with round apertures are used for camera modules with small focallengths, e.g., an equivalent a focal length shorter than 70 mm fullframe DSLR. Focal length of a lens (camera) is often stated as theequivalent focal length for a full frame DSLR camera where a DSLR camerais digital single-lens reflex camera. A lens or system such as anoptical chain with an equivalent focal length of 35 mm will frame thesame shot from the same location as a full frame DSLR camera with a 35mm lens would frame. The actual focal length of the optical chain withan equivalent focal length of 35 mm may be significantly smaller becausethe sensor is typically much smaller than a full frame DSLR sensor. Ingeneral if the sensor is 5 times smaller in each dimension (25 timessmaller area), an optical chain of 7 mm real focal length will have afocal length equivalent to 35 mm of a full frame DSLR.

While in various embodiments light redirection is used for cameramodules with some focal lengths to allow for a thin cameraimplementation, some other camera modules with shorter, e.g., a focallength equivalent to a 35 mm full frame DSLR focal lengths, maybe andsometimes are implemented without the use of light redirection elements.

In some embodiments, to allow for camera thickness to be less than themaximum dimension of the outermost lens element of a camera module(which is along the horizontal direction in some embodiments) andcorresponding aperture, a light redirection device, which is configuredto support a bigger aperture in one direction, e.g., the horizontaldirection than another direction, e.g., the vertical direction, is usedin some embodiments. This light redirection device redirects lighttraveling towards the camera along the optical axis of the camera moduleto travel in a different direction, e.g., vertically, after redirection.This allows the depth of the light redirection device to be smaller thanthe maximum dimension of the aperture.

The use of a redirection element which supports a bigger aperture in onedimension (e.g., horizontal) than the other results in the capture of atwo dimensional image with a higher quality, e.g., more captured higherfrequency image content, in one dimension (e.g., horizontal) than theother. While such an image may be less desirable than a more uniformimage where the same amount of optical information is captured in boththe vertical and horizontal dimensions, the thinner camera width madepossible by using a light redirection element which-supports an oblongaperture can be desirable because it allows the camera to be thinnerthan the longest length of the lens opening though which light entersthe camera. Thus, in such an embodiment the thickness of the camera isnot coupled or constrained by the maximum dimension of an oblong ornon-circular lens opening (aperture).

The use of non-circular, e.g., oblong, oval or other non-circular,apertures can result in more high frequency image information beingcaptured in one dimension, e.g., along the longer dimension of thenon-circular lens, than in the other dimension. In various embodimentsto make up for the difference in high frequency information in differentdirections resulting from a lens and/or effective aperture beingnarrower in one dimension than the other, multiple optical chains withdifferent orientations of the higher resolution dimension are used. Theoptical chains with different orientations capture images with differentdirections (e.g., horizontal, vertical, slanted) of the highestresolution dimension in the image, where the camera is in a verticalorientation and the camera lenses are facing forward.

In one such embodiment, the images captured by the different opticalchains are combined to form a combined or composite image. As should beappreciated by combining images with different amounts of high frequencyinformation in different dimensions, e.g., captured by different opticalchains at the same time, the combined image need not be constrained bythe lower frequency information captured in one dimension by an opticalchain with a non-round aperture stop. This is because the image capturedby a different optical chain, e.g., an optical chain with an orientationallowing for capture of higher frequency information in the dimensionwhere the other optical chain suffers, can be used to make up for theweakness of the image captured by the other optical chain. As an exampleif there are two optical chains both having non-round, e.g., oblong,apertures, the orientation of the first optical chain maybe such thatthe aperture of the first optical chain is larger in the horizontaldimension giving the image captured by the first optical chain a higherresolution in the horizontal direction, while the second optical chainorientation maybe such that it correspondingly captures images with ahigher vertical resolution. The two images can, and in some embodimentsare, combined into a single composite image that has the higherresolution in both vertical and horizontal directions.

Thus, by using multiple optical chains with non-round apertures butdifferent orientations, and combining the images captured by suchoptical chains, a relatively thin camera can be implemented without thequality (sharpness) of the image being constrained by the use of asmaller round aperture the diameter of which would normally be limitedby the camera depth.

In some embodiments non-round lenses are used as the outermost lens ofmultiple optical chains used to capture images in parallel which arethen combined to form a combined image. The non-round lenses may beformed by cutting off portions of a round lens, e.g., left and rightportions, to form an oblong lens. In other embodiments portions of roundlenses are masked by an applied mask or a portion of the camera housingin which the lenses are mounted forming lenses with oblong apertures. Instill other embodiments round outer lenses are used and the lightredirecting device is of a size and shape such that not all the lightpassing through the round lenses will be redirected to the sensor. Insuch embodiments the light redirection device operates as the constrainton the optical light path and becomes the aperture stop, e.g., the pointof light constraint, on the light being passed to the sensor. Thus, invarious embodiments the oblong lens serves as the aperture stop with theoblong aperture being the point of constriction on the light path of theoptical chain while in other embodiments the light redirection elementservers as a constraint on the light path and the aperture stop in theoptical chain with the effective aperture being oblong in shape despitethe fact that the outer lens may have been round.

Methods and apparatus which use multiple optical chains to capturemultiple images of an area at the same time are described. The multiplecaptured images may, and in some embodiments are then combined to form acombined image. The combined image in various embodiments is normally ofhigher quality than would be achieved using just a single one of theoptical chains.

Various embodiments, provide many of the benefits associated with use ofa large lens and/or large high quality sensor, through the use ofmultiple optical chains which can normally be implemented using smallerand/or lower cost components than commonly used with a high qualitylarge lens single optical chain camera implementation.

In various embodiments an optical chain, e.g., camera module, includes acombination of elements including one or more lenses, a lightredirection device and a sensor. The light redirection device is a lightdiverter and may take various forms, e.g., it may be a mirror or prism.The light redirection device may be hinged to allow the angle and thusdirection in which an optical chain is pointing to be changed by movingthe light redirection device.

In at least some embodiments images captured by different optical chainswith non-round apertures having different orientations are combined. Insome embodiments the images from two, three or more, e.g., six or more,optical chains with different orientations are combined to form a singlecombined image. While images from optical chains with differentorientations are combined in some embodiments, it should be appreciatedthat images captured by more than one optical chain with the sameorientation can be combined with one or more images captured by opticalchains with a different orientation, e.g., relative to the bottom of thecamera, e.g., the horizontal, for purposes of explanation. Thus, bycombining images from different optical chains many advantages can beachieved allowing for multiple small lenses to be used and a relativelythin camera housing as compared to systems using a single large roundlens.

In various embodiments the outer lens of the multiple optical chains arefixed and thus unlike many conventional zoom camera devices in suchembodiments the outer lenses, i.e., the lenses on the face of thecamera, do not move out of the camera body and are fixed with respect tothe face of the camera even during zoom operations. The outermost lensesmay, and in some embodiments do have zero or very little optical powerand serve largely as a cover to keep dirt out of the optical chains towhich the outer lens corresponds. The outer lens in such embodiments maybe implemented using flat glass or plastic. In some embodiments aslideable cover is slide over the outer lenses when the camera is to beplaced in storage and slide back when the camera device is to be used.FIG. 14 shows one such embodiment with the lenses being uncovered andthe cover slide to a position in which the case which includes the lenscover can be used as a camera grip or handle.

In some embodiments while a portion of the outermost lens may extendfrom the front of the camera device beyond the surface of the cameradevice, the outermost lenses generally extend, if at all, a small amountwhich is less than the thickness of the camera. Thus even during use thelenses to not extend significantly beyond the face of the camera devicein which the optical chains are mounted and normally less than half thethickness of the camera device at most.

In many if not all cases images representing real world objects and/orscenes which were captured by one or more of the optical chain modulesof the camera device used to take the picture are preserved in digitalform on a computer readable medium, e.g., RAM or other memory deviceand/or stored in the form of a printed image on paper or on anotherprintable medium.

While explained in the context of still image capture, it should beappreciated that the camera device and optical chain modules of thepresent invention can be used to capture video as well. In someembodiments a video sequence is captured and the user can select anobject in the video sequence, e.g., shown in a frame of a sequence, as afocus area, and then the camera device capture one or more images usingthe optical chain modules. The images may, and in some embodiments are,combined to generate one or more images, e.g., frames. A sequence ofcombined images, e.g., frames may and in some embodiments is generated,e.g., with some or all individual frames corresponding to multipleimages captured at the same time but with different frames correspondingto images captured at different times.

Different optical chain modules maybe and sometimes are controlled touse different exposure times in some embodiments to capture differentamounts of light with the captured images being subsequently combined toproduce an image with a greater dynamic range than might be achievedusing a single exposure time, the same or similar effects can and insome embodiments is achieved through the use of different filters ondifferent optical chains which have the same exposure time. For example,by using the same exposure time but different filters, the sensors ofdifferent optical chain modules will sense different amounts of lightdue to the different filters which allowing different amount of light topass. In one such embodiment the exposure time of the optical chains iskept the same by at least some filters corresponding to differentoptical chain modules corresponding to the same color allow differentamounts of light to pass. In non-color embodiments neutral filters ofdifferent darkness levels are used in front of sensors which are notcolor filtered. In some embodiments the switching to a mode in whichfilters of different darkness levels is achieved by a simple rotation ormovement of a filter platter which moves the desired filters into placein one or more optical chain modules.

The camera devices of the present invention supports multiple modes ofoperation and switching between different modes of operation. Differentmodes may use different numbers of multiple lenses per area, and/ordifferent exposure times for different optical chains used to capture ascene area in parallel. Different exposure modes and filter modes mayalso be supported and switched between, e.g., based on user input.

Numerous additional variations and combinations are possible whileremaining within the scope of the invention.

The techniques of the present invention may be implemented usingsoftware, hardware and/or a combination of software and hardware. Thepresent invention is directed to apparatus, e.g., mobile nodes such asmobile terminals, base stations, communications system which implementthe present invention. It is also directed to methods, e.g., method ofcontrolling and/or operating mobile nodes, base stations and/orcommunications systems, e.g., hosts, in accordance with the presentinvention. The present invention is also directed to machine readablemedium, e.g., ROM, RAM, CDs, hard discs, etc., which include machinereadable instructions for controlling a machine to implement one or moresteps in accordance with the present invention.

In various embodiments devices described herein are implemented usingone or more modules to perform the steps corresponding to one or moremethods of the present invention, for example, control of image captureand/or combining of images. Thus, in some embodiments various featuresof the present invention are implemented using modules. Such modules maybe implemented using software, hardware or a combination of software andhardware. In the case of hardware implementations embodimentsimplemented in hardware may use circuits to as modules alone or incombination with other hardware elements. Many of the above describedmethods or method steps can be implemented using machine executableinstructions, such as software, included in a machine readable mediumsuch as a memory device, e.g., RAM, floppy disk, etc. to control amachine, e.g., a camera device or general purpose computer with orwithout additional hardware, to implement all or portions of the abovedescribed methods, e.g., in one or more nodes. Accordingly, among otherthings, the present invention is directed to a machine-readable mediumincluding machine executable instructions for causing or controlling amachine, e.g., processor and associated hardware, to perform one or moreof the steps of the above-described method(s).

While described in the context of an cameras, at least some of themethods and apparatus of the present invention, are applicable to a widerange of image captures systems including tablet and cell phone deviceswhich support or provide image capture functionality.

Images captured by the camera devices described herein may be real worldimages useful for documenting conditions on a construction site, at anaccident and/or for preserving personal information whether beinformation about the condition of a house or vehicle.

Captured images and/or composite images maybe and sometimes aredisplayed on the camera device or sent to a printer for printing as aphoto or permanent document which can be maintained in a file as part ofa personal or business record.

Numerous additional variations on the methods and apparatus of thepresent invention described above will be apparent to those skilled inthe art in view of the above description of the invention. Suchvariations are to be considered within the scope of the invention. Invarious embodiments the camera devices are implemented as digitalcameras, video cameras, notebook computers, personal data assistants(PDAs), or other portable devices including receiver/transmittercircuits and logic and/or routines, for implementing the methods of thepresent invention and/or for transiting captured images or generatedcomposite images to other devices for storage or display.

What is claimed is:
 1. A method of capturing images using a cameradevice including at least a first optical chain including a first sensorand a second optical chain including a second sensor, said first opticalchain having a smaller focal length than said second optical chain, themethod comprising: making a decision based on at least one of: i) userinput or ii) a determined rate of motion whether to control the secondsensor of the second optical chain to perform joint readouts of pixelelements corresponding to consecutive pixel element rows of the secondsensor or to perform individual read outs of consecutive rows of pixelelements of said second sensor; controlling the first sensor to readoutpixel values of said first sensor; and controlling the second sensor toreadout pixel values of pixel elements of said second sensor inaccordance with the said decision to perform joint readouts of pixelelements corresponding to consecutive pixel element rows of the secondsensor or to perform individual read outs of consecutive rows of pixelelements of said second sensor.
 2. The method of claim 1, wherein saidfirst focal length is smaller than said second focal length.
 3. Themethod of claim 1, wherein controlling the second sensor to readoutpixel values involves controlling the second sensor to readout pixelvalues in parallel with the readout of at least some pixel values ofsaid first sensor.
 4. The method of claim 1, wherein said step of makinga decision is based on user input, said user input indicting a high rateof motion mode of operation or a high resolution mode of operation, saidstep of making a decision is based on user input including: i) decidingto perform joint readouts of pixel elements corresponding to consecutivepixel element rows of the second sensor when the user input indicatesselection of a high rate of motion mode of operation; and ii) decidingto perform individual read outs of consecutive rows of pixel elements ofsaid second sensor when the user input indicates selection of a highresolution motion mode of operation.
 5. The method of claim 3, whereinsaid step of making a decision is based on a rate of motion, said stepof making a decision including: i) deciding to perform joint readouts ofpixel elements corresponding to consecutive pixel element rows of thesecond sensor when the rate of motion is above a first threshold; andii) deciding to perform individual read outs of consecutive rows ofpixel elements of said second sensor when the rate of motion is belowsaid first threshold.
 6. The method of claim 5, further comprising:determining said rate of motion from images sequentially captured by anoptical chain of said camera device.
 7. The method of claim 6, whereinsaid determining includes determining the rate of motion based ondifferences between an image captured during a first time period and animage captured during a second time period which follows said first timeperiod.
 8. The method of claim 1, further comprising: controlling athird sensor corresponding to a third optical chain having the samefocal length as said second optical chain to perform joint readouts ofpixel elements corresponding to consecutive pixel element rows when saiddecision is a decision to perform joint readouts of pixel elementscorresponding to consecutive pixel element rows of the second sensor,said third sensor being controlled to jointly read out at least somerows of pixel values after the readout of said second sensor has beencompleted but prior to completion of the readout of the first sensor. 9.The method of claim 1, further comprising: when said decision is adecision to perform joint readouts of pixel elements corresponding toconsecutive pixel element rows of the second sensor, controlling thenumber of rows of pixel elements to be read out in a joint read as afunction of the ratio of the first focal length to the second focallength.
 10. The method of claim 1, further comprising: controlling thefirst sensor to readout pixel values of pixel elements of said firstsensor at a rate which is a function of the ratio of the second focallength to the first focal length when said decision is a decision toperform individual read outs of consecutive rows of pixel elements ofsaid second sensor.
 11. A camera device comprising: a first opticalchain including a first sensor and having a first focal length; a secondoptical chain including a second sensor and having a second focallength, said first focal length being smaller than said second focallength; a readout control decision module configured to make a readoutcontrol decision, based on at least one of: i) user input or ii) adetermined rate of motion, whether to control the second sensor of thesecond optical chain to perform joint readouts of pixel elementscorresponding to consecutive pixel element rows of the second sensor orto perform individual read outs of consecutive rows of pixel elements ofsaid second sensor; and a sensor readout control module configured to:i) control the first sensor to readout pixel values of said firstsensor; and ii) control the second sensor to readout pixel values ofpixel elements of said second sensor in accordance with the said readoutcontrol decision.
 12. The camera device of claim 11, wherein said firstfocal length is smaller than said second focal length.
 13. The cameradevice of claim 11, wherein said sensor readout control module controlsthe second sensor to readout pixel values in parallel with the readoutof at least some pixel values of said first sensor, as part ofcontrolling the second sensor to readout pixel values.
 14. The cameradevice of claim 11, further comprising: a user input interfaceconfigured to receive user input indicating an automatic mode ofoperation, a high rate motion mode of operation or a high resolutionmode of operation; a mode control module configured to set the cameradevice in a mode of operation in response to received user input; andwherein said readout control decision module is configured to make saiddecision based on a rate of motion detected in images captured by saidcamera device when received user input indicates an automatic mode ofoperation.
 15. The camera device of claim 13, wherein said readoutcontrol decision module is configured to i) decide to perform jointreadouts of pixel elements corresponding to consecutive pixel elementrows of the second sensor when the rate of motion is above a firstthreshold; and ii) decide to perform individual read outs of consecutiverows of pixel elements of said second sensor when the rate of motion isbelow said first threshold.
 16. The camera device of claim 15, furthercomprising: a motion rate determination module configured to determinesaid rate of motion from images sequentially captured by an opticalchain of said camera device.
 17. The camera device of claim 16, whereinsaid motion rate determination module determines the rate of motionbased on differences between an image captured during a first timeperiod and an image captured during a second time period which followssaid first time period.
 18. The camera device of claim 11, wherein thesensor readout control module is further configured to: control a thirdsensor corresponding to a third optical chain having the same focallength as said second optical chain to perform joint readouts of pixelelements corresponding to consecutive pixel element rows when saiddecision is a decision to perform joint readouts of pixel elementscorresponding to consecutive pixel element rows of the second sensor,said third sensor being controlled to jointly read out at least somerows of pixel values after the readout of said second sensor has beencompleted but prior to completion of the readout of the first sensor.19. The camera device of claim 11, further comprising: a joint readoutcontrol module configured to control the number of rows of pixelelements to be read out in a joint read as a function of the ratio ofthe first focal length to the second focal length when said decision isa decision to perform joint readouts of pixel elements corresponding toconsecutive pixel element rows of the second sensor.
 20. Anon-transitory machine readable medium including processor executableinstructions which when executed by a processor of a camera deviceincluding at least a first optical chain including a first sensor and asecond optical chain including a second sensor, said first optical chainhaving a smaller focal length than said second optical chain, controlthe camera device to perform the steps of: making a decision based on atleast one of: i) user input or ii) a determined rate of motion whetherto control the second sensor of the second optical chain to performjoint readouts of pixel elements corresponding to consecutive pixelelement rows of the second sensor or to perform individual read outs ofconsecutive rows of pixel elements of said second sensor; controllingthe first sensor to readout pixel values of said first sensor; andcontrolling the second sensor to readout pixel values of pixel elementsof said second sensor in accordance with the said decision to performjoint readouts of pixel elements corresponding to consecutive pixelelement rows of the second sensor or to perform individual read outs ofconsecutive rows of pixel elements of said second sensor.